SEMISIMPLE ORBITAL INTEGRALS ON THE SYMPLECTIC SPACE 
FOR A REAL REDUCTIVE DUAL PAIR 



M. MCKEE, A. PASQUALE, AND T. PRZEBINDA 

Abstract. We prove a Weyl Harish-Chandra integration formula for the action of a 
reductive dual pair on the corresponding symplectic space W. As an intermediate step, 
we introduce a notion of a Cartan subspace and a notion of an almost semisimple element 
in the symplectic space W. We prove that the almost semisimple elements are dense in 
W. Finally, we provide estimates for the orbital integrals associated with the different 
Cartan subspaces in W. 



Contents 

1. Introduction 1 

2. Intertwining distributions 4 

3. Singular semisimple orbital integrals on a semisimple Lie algebra 6 

4. A Weyl Harish-Chandra formula on the odd part of an ordinary classical Lie 
super algebra 13 

5. A semisimple orbital integral on the odd part of an ordinary classical Lie 
superalgebra 26 

6. Some properties of the invariant eigendistributions on the symplectic space. 31 
Appendix A 34 
Appendix B 35 
Appendix C 51 
Appendix D 53 
References 53 



1. Introduction 

Let W be a symplectic real vector space, Sp the corresponding symplectic group and 
Sp the metaplectic group. Let G, G' C Sp be a real reductive dual pair, and let G and 
G be the preimages of G and G in Sp, respectively. Furthermore, let 11 ® H' be an 
irreducible admissible representation of G x G' in Howe's correspondence. Each such 
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representation 11 (g) 11' is attaclicd to a tempered distribution / = /n^n' on W, called the 
intertwining distribution of H (8) H', which is uniquely determined up to a scalar multiple. 
One may therefore expect that the properties of the intertwining distribution contain 
useful information on the representation itself. 

Let Q and g' respectively denote the Lie algebras of G and G'. Let U{q) denote the 
universal enveloping algebra of g, and let U{g,)'^ be the subalgebra of the G- invariants in 
U{q). Then the intertwining distribution turns out to be G x G'- invariant and an eigendis- 
tribution of and U{q')'^ , with eigenvalue equal to the infinitesimal characters of 11 

and n', respectively; see [Prz93]. In other words, /mgin' is an invariant eigendistibution 
on the symplectic space W. 

Harish-Chandra's method of descent is one of the main tools for studying questions 
involving the structure of invariant eigendistributions on a real reductive Lie algebra g. 
Ultimately, it transfers problems of invariant harmonic analysis from the Lie algebra g to 
the Cartan subalgebras of q, see for example [Bou94]. Our goal is to develop a "method of 
descent" allowing us to study invariant eigendistributions on symplectic spaces. We use 
the fact that the symplectic space W is the odd part Sj of a classical real Lie superalgebra 
s, which is constructed from the dual pair (G, G'). The method of descent will consist in 
transferring problems of invariant harmonic analysis from s- to suitably defined Cartan 
subspaces in Sj. (Notice that the name "Cartan subspace" is also used in other contexts, 
for example in the theory of symmetric spaces. Our usage of this term is however different.) 

To single out the candidates for the Cartan subspaces in Sj, one needs a detailed 
knowledge of the geometry of the adjoint action of S = G x C on Sj. This step has 
been accomplished in [Prz06] for all irreducible reductive dual pairs except for the two 
following ortho-symplectic cases: 



The results of this paper also include these two cases. This leads us to a definition of 
Cartan subspace which is slightly different from the one used in [Prz06], as well as to the 
notion of almost semisimple elements in Sj. The definitions agree with those from [Prz06] 
when the dual pair is not isomorphic to (1) or (2). In fact, most of the results of [Prz06] 

carry over. 

Our first result is an analog of the Weyl Harish- Chandra formula on the symplectic 
space W (Theorem 20). It gives the integral on W = of a continuous compactly 
supported function in terms of semisimple orbital integrals parametrized by the mutually 
non-conjugate Cartan subspaces of s-. As an important intermediate step, we prove that 
the set of the semisimple elements in W is dense, unless the dual pair is isomorphic to 
(1) or (2). This property was stated in [Prz06, Proposition 6.6] without a proof. Here 
we show explicitly how any nilpotent element of W may be approximated by semisimple 
elements. For the remaining cases we introduce the notion of almost semisimple elements 
and show that they are dense (Theorem 19). 




(1) 
(2) 



Op(C) X Sp2„(C) with p odd and p < 2n. 
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Weyl Harish-Chandra formula provides an orbital integral decomposition for the inte- 
gral on\N — Sj of continuous compactly supported functions. For the analysis of tempered 
eigendistributions, one needs to know that this formula extends to Schwartz functions on 
W. The main result of this article, Theorem 25, shows that any such orbital integral 
of a rapidly decreasing function on W is rapidly decreasing at infinity on though it 
might have logarithmic growth near walls defined by some real roots. This guarantees the 
extension of Weyl Harish-Chandra formula to Schwartz functions. However, unlike the 
Lie algebra case, it is not necessarily true that a semisimple orbital integral of a Schwartz 
function on the symplectic space is a Schwartz function on the corresponding Cartan sub- 
space, though it is certainly differentiable on the regular set. Since the radial components 
of invariant differential operators in this setting do not behave as well as in the case of a 
Lie algebra, it doesn't seem to be easy to estimate these derivatives. 

Our analysis follows closely Harish-Chandra's work. Indeed, the even part % of the 
Lie superalgebra s is the reductive Lie algebra Q x q'. So the map of Sj to % given by 
z z'^ relates geometrical and analytical objects on Sj to the corresponding objects on 
So — 5 X q'- In particular, if f)Y C sj is a Cartan subspace, then [)p that is the linear 
span of the all the anticommutants of the elements of is contained in some Cartan 
subalgebra of %. Nevertheless we shall need several ingredients not included in the work of 
Harish-Chandra. For instance, in the proof of Proposition 5, we need Rossmann's theorem 
expressing some nilpotent orbital integrals as limits of derivatives of the Harish-Chandra 
semisimple integrals, [Ros90]. 

The regularity properties of an invariant eigendistribution / on W = can be obtained 
from the analysis of the system of partial differential equations it satisfies. By determining 
the characteristic variety of this system, we prove in Corollary 28 that the restriction of 
/ to the set C W of the regular almost semisimple elements is a smooth function. 

However, most of the times, this distribution is not regular on the entire W. We will 
provide some examples in our future work. 

Our paper is organized as follows. In section 2 we recall the construction of the in- 
tertwining distribution attached to an irreducible admissible representation in Howe's 
correspondence. In section 3 we collect some results on orbital integrals on semisimple 
Lie algebras which will be needed in the Lie superalgebra situation. The study of the 
adjoint action of a dual reductive pair G x G' on the symplectic space W is developed 
in section 4. We use the Lie superalgebra approach from [Prz06] by considering W as 
the odd part Sj of a classical real Lie superalgebra s. We introduce a general notion of 
Cartan subspace of Sj which for complex dual pairs coincides with the notion of Cartan 
subspace in [DK85] . Then we prove the density in Sj of the almost semisimple elements 
(Theorem 19). This yields the Weyl Harish-Chandra formula on Sj (Theorem 20). In 
section 5 we make a detailed study of the orbital integrals through each Cartan subspace 
in Sj. Theorem 25 provides sharp estimates for the orbital integral of rapidly decreasing 
functions on W. The structure theory developed in section 4 reduces the proof of this 
theorem to the analysis of three special cases: the case of elliptic Cartan subspaces; the 
case a dual pair of the form (GLi(D), GLi(D)), where D = IR, C or H (the quaternions); 
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the case of (GL2(M), GL2(M)) with centrahzer of [)j in Sq equal to a fundamental Cartan 
subalgebra. Finally, in section 6 we include a brief study of the regularity properties of 
the invariant eigendistributions from the differential equation point of view. 

Some proofs that have not been included in the main text have been collected in the 
Appendices A,B, C and D. 

2. Intertwining distributions 

Let W be a vector space of finite dimension 2n over R with a non-degenerate symplcctic 
form (■, ■). Denote by Sp the corresponding symplectic group and by sp its Lie algebra. 
Recall the Cayley transform c{y) = {y + l){y — 1)"^, [How88], and let 

S^' = = (g, e Sp X C, detig -1)^0, e = det{i{g - 1))-'}. (3) 

For each x G sp, {x-, ■) is a symmetric bilinear form on W with the signature sgn(a;-, ■) 
equal to the maximal dimension of a subspace where this form is positive definite minus 
the maximal dimension of a subspace where this form is negative definite. Set 

chc{x) = 2""! det(x)| 2 exp(— i sgn{x-, •)) {x G 5p, det(a;) ^ 0). (4) 

(This is a Fourier transform of one of the two minimal non-zero nilpotent co-adjoint 
orbits in sp*, [PrzOO, Proposition 9.3] and [Hor83, Theorem 7.6.1].) For two elements 
(fl'i)Ci)) {92,^2} G Sp , with c{gi) + c{g2) invertible, define the product 

(^1, ^i)(^2, 6) = {9192, iii2chc{c{gi) + c{g2))). (5) 

Theorem 1. [HowSS, sec. 16] Up to a group isomorphism there is a unique connected 
group Sp containing Sp with the multiplication given by (5) on the indicated subset of 
Sp X Sp . The map 

Sp 9 ^ H-^ 5( e Sp 
extends to a double covering homomorphism 

Sp 3 g ^ g e Sp. 

Fix the unitary character x(r) = e'^'^^^, r G M, and a suitably normalized Lcbesgue 
measure dw on W. For 0i,02 G S{\N), the Schwartz space of W, we have the twisted 
convolution, [HowSS], 

MMw') = [ (piMMw' -w)x{l{w,w'))dw (w'gW). (6) 

Multiplication by the Lebesgue measure gives an embedding of S{\N) into yS'*(W), the 
space of the temperate distributions on W. The twisted convolution (6) extends to some, 
but not all, temperate distributions. Also, we have a ^-operation on S'(W), 

= (w G W,0 G 5(W)), (7) 

which does extend to S*{\N) by 

r (0) = fir) if eS*(\N),<t>e S(\N)). (S) 
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Define the following functions 

where, for x — c{g) e sp, we have set Xxi"^) — x{\{^{'^)-i'^))- 

Theorem 2. [How88] The map T extends to an injective continuous map T : Sp ^ S*{\N) 
and the following formulas hold 

T{g,)^T{g2) = T{gr~g2) {gi^h G Sp', det(c(£i) + 0(52)) ^ 0), 

T{gy = T{g-') e Sp), 

T(l) = 5, 

where S is the Dirac delta at the origin. 

The oscillator representation of Sp may be realized as follows. Pick a complete polar- 
ization 

W = X©Y (9) 

and recall the Weyl transform 

JC: S*{\N) ^ S*{XxX), (10) 

/C(/)(x, x') ^J^f(x-x' + y)x(l{y, X + x')) dy, f e S(\N) 
with the inverse given by 

}C-\K)(x + y)^2-^ J^K{^^,^^)x(l{x',y))dx' (x,x' eX, yeY). (11) 
Each element K eS*{XxX) defines an operator OP{K) e Hom(5(X), S*{X)) by 



OP{K)v{x)= I K{x,x')v{x')dx'. (12) 
ix 



The map OP extends to an isomorphism of linear topological spaces, S*(X x X) and 
Hom(5'(X), 5'*(X)). This is known as the Schwartz Kernel Theorem, [Hor83, Theorem 
5.2.1]. _ 

The oscillator representation u of Sp, corresponding to the character x, is defined by 

uj{g) = OP{IC{T{g))) {g G S^). (13) 

Each operator (13) extends to a unitary operator on the Hilbert space Hu, = L'^iX) and to 
an automorphism of S*{X). The space of the smooth vectors l-f^ = S{X), and coincides 

with the distribution character of a;. _ _ 

Let G, G' C Sp be a dual pair. Denote by G, G' the preimages of G and G' in Sp. Let 
n (8) n' be an irreducible admissible representation of G x G' in Howe's correspondence. 
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Then, up to infinitesimal equivalence, 11 ® 11' may be realized as a subspace of "H^* = 
S*{X). Furthermore, Howe proved in [How89] that 

dim(Homgg,(?^^, n ® n')) = 1. (14) 

Thus for each representation 11(8)11' there is a unique (up to a scalar multiple) distribution 
/n(g)n' G S*{\N), called the intertwining distribution, such that 

C • OP(/C(/n^n')) = Homgg,(?^~, U ® H')). (15) 

In this exposition we presented the Schrodingcr model of the oscillator representation 
cu attached to the polarization (9), but the distribution / = /n®n' does not depend on 
this particular model. Furthermore, as shown in [Prz93], the intertwining distribution / 
satisfies the following system of differential equations: 

T{z)[^f = ^n{z)f (zeUidf), (16) 

where 

T{z)^f = jT{eMtz))mt=o {z e 0) 

and 7n : 1^{q)'^ — > C is the infinitesimal character of 11. (One obtains the same system of 
equations replacing U{g)'^ by U{q')'^' in (16).) Thus, / is an invariant eigendistribution 
on the symplectic space W. 



3. Singular semisimple orbital integrals on a semisimple Lie algebra 

This section collects some properties which will be needed in section 5 for studying the 
convergence of semisimple orbital integrals on a symplectic space. These properties adapt 
to our situation some well known results of Harish-Chandra; see [Har57b, sections 4 and 
5] or [Var77, Part 1, Ch. 3]. 

As in [Har57b], Q stands for a real semisimple Lie algebra and G is the (connected) 
adjoint group of q. Let f) C g be a Cartan subalgebra. Fix an element z Let 3 = 0^ 
denote the centralizer of 2; in g and let Z C G be the ccntralizer of z in G. Then Z is a 
real reductive group with the Lie algebra 3. Let c denote the center of 3. Then c C f) C 3. 
Recall that a root a for the pair (()c,Sc) is said to be real if a{{}) C R and imaginary if 
aibi) C iM; if a is neither real nor imaginary, it is said to be complex. We fix a set of 
positive roots of (l)ci0c)- Foi' 3 ^ 1 ^ 0i let VTg/q denote the product of all positive roots 
such that the corresponding root spaces do not occur in qc. Thus TTg/t, is the set of all 
positive roots and 

^0/3= n ^- ^^'^) 

a>0,a(z)7^0 

Let c^^^ C c be the subset where tTj/j does not vanish. Moreover, denote by d{gZ) a positive 
G- invariant measure on the quotient space G/Z. This measure does exist because both 
G and Z are unimodular. 
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In the proofs of this section we shall need some notation concerning constant coefficients 
differential operators on g. For x e let d{x) denote the corresponding directional 
derivative 

d{x)^l;{y)^j^^l;{y + tx)\t=o {x,y e Q, e ^-(g)) . (18) 

Then d extends to an isomorphism from the symmetric algebra S{q) of Qc onto the 
algebra of the constant coefficient differential operators on q. If m G S{q) is homogeneous 
of degree d, then the differential operator d{u) is said to be of degree d. Fix a ^-invariant 
non-degenerate symmetric bilinear form B(-, •) on the real vector space Q. (So S is a 
nonzero constant multiple of the Killing form.) This form extends to the complexification 
Qc and provides an isomorphism of Qc with the dual g^, and hence a Gc-equivariant 
identification of 5(g) with V{q), the algebra of the complex valued polynomial functions 
on g. Here Gc denotes the adjoint group of gc- Furthermore, let D be a differential 
operator on g and let x e g. Then there exists a unique element p E S{g) such that 

D^l^ix) ^ {d(p)^l;)(x) (V'eC~(g)). 

The constant coefficient differential operator d{p) is called the local expression of D at x, 
and is denoted by D\x. 

In the following, | ■ | is a fixed norm on the real vector space g and g.x denotes the 
adjoint action of gi e G on x e g. 

Theorem 3. Suppose z E i) is not annihilated by any real or complex root of f) in gc- 
Then for every M > there is N > such that 

sup (1 + |x|)^|7rg/3(x)| / (1 + \g.x\)-''d{gZ) < oo. 
lec^s Jg/z 

Proof. Assume first that the Cartan subalgebra f) is elliptic, so that the corresponding 
Cartan subgroup H C G is compact. 

Let 3" = [3,3] be the derived algebra of 3 and let b = \j f] 3". Then 3 = c © 3" and 
f) = c© b. Moreover, b is an elhptic Cartan subalgebra of 3". Let Tr^/f, denote the product 
of all the positive roots for ([)c, gc) such that the corresponding root spaces occur in jc- 

According to Harish-Chandra, [Har64, Theorem 3, page 568], there is a non-zero con- 
stant C eR such that for all / e C^{$) and all x e c, 

-Cf{x). (19) 



\ '^ih{x + y) / f{x + g".y)d{g"ll 
\ Jzin 



)) 
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Here y E b, 7i^/i,{x + y) = TCj/tjiy) and the differential operator d{7!'^/t,) acts with respect to 
the variable y. Let ijj E C^(q). Then (19) shows that for x E c''^^, 



'G/H 



(20) 



y=o 



dm 



^3/^(^ + y) / Hg-i^ + 9"-y)) d{g"m 

G/Z \ Jz/H 

^ C ! ij{g.x)d{gZ). 

JG/Z 

Let 1^(0, [)) denote the Wcyl group of ({)C)0c) and let p(f))^(0''') be the space of the 
iy(0, {))-invariants in V{\)). Let rj e V{i))^^^'^^ be viewed as a differential operator of 
degree zero, i.e. multiplication by rj. Then 



(21) 



where the left hand side is the local expression at zero of the composition of the two 
differential operators and 7/(0) G C is the value of r/ at G f). Indeed, there is a unique 
C G 'P({)) such that the left hand side of (21) is equal to d{(). Since TTg/^ is W{q, l))-skew 
invariant, so is (. Lemma 10 in [Har57a, page 100] implies that there is ^ G 7'(f))^(S'^) 
such that ( = ^vTg/t,. But, by the definition of the degree of ( is less or equal to the 
degree of TTg/f,. Hence ,^ is a constant. 

Let f]~^ be the sum of the homogeneous components of positive degrees of rj, so that 77 ~ 
77(0) +77+. Then 9(7rg/f,)(77+7r,/^)(0) = 0. Hence, d{n,/^){r]7f,/^m = rj{0){d{7r,/^)7r,/^){0). 
Therefore, 

C{dMn,/^){0) = a(C)7rg/^(0) = d{7:,/^){rj7r,/^m = v{0){d{7i,/^)7T,/^){Q), 

which shows that ^ = ri{0), and (21) follows. 

We apply (21) with g replaced by 3 and rj replaced by TTg/j, which is W{^, [)) -invariant. 
Then for x G c''^^ and ye i)", 



y=o 



y=0 



(22) 



Since TTg/i, = ''^B/}''^}/t)j equations (20) and (22) show that for ip G C^{q) and x G c'''^^. 



Jg/h 



y))d{gli) 



0/3 



.X 



y=o 



i,{g.x)d{gZ). (23) 



G/Z 



Since C 7^ 0, (23) together with a theorem of Harish-Chandra, [Har57b, Theorem 2, page 
207 and Lemma 25, page 232] imply that for any M > there is a continuous seminorm 
V on S{q) such that for all -0 G C'^{q) and all x G c*"^^ 



[l + \x\ 



7^8/3 (^) / ij{9-x)d{gZ) 



G/Z 



(24) 
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Notice that for each x G c*""^^ the formula 



[l + \x\)''Mx)\ [ i^{g.x)d{gZ) 
Jg/z 



G/Z 

defines a positive Borel measure on q. Hence, [Var77, Lemma 8, page 37] shows that there 
are finite non-negative constants C, N such that 

(i + Nrk/3(^)l / (i + |^.x|)-^%z)<c (xec-^). (25) 
Jg/z 

Thus our Theorem is vahd for any elhptic Cartan subalgebra f) C 

Now we drop the assumption that {) is elhptic. Fix a Cartan involution ^ on g and let 
= ^ © p be the corresponding Cartan decomposition. We may assume that f) is ^-stable. 
Then f) = t © a, where t = fl f and a = ^Hp. Let t = 0" C g be the ccntralizer of a and 
let m be the orthogonal complement (with respect to the form B{9-, ■)) of a in (. Then m 
is reductive, ( = m © a and t is an elhptic Cartan subalgebra of m. Fix a set of positive 
roots of a in and let n C g denote the sum of all the corresponding root spaces. Then 
(see [Har57b, page 216]) 

kfl/f,(a;)| = I det(ad x)n\\n^/i{xt)\ {x e f)), (26) 

where x — Xi + Xa, Xi & t, Xa & a. Furthermore, if x e f) is such that no real roots vanish 
on X, then with the appropriate normalization of all the invariant measures involved. 



Ig, 



IG/A 

where 



1 f ~ 

ibiq.x) d(qA) = — — ; — ; — — - / ih(m.x) dm (27) 
^ ^ I det(ad x)„| 7m 



Hy)= [ [ i^{k.{y + X))dkdX {yei). 

Jn J K 

(See [Har57b, Corollary on page 217] or [Var77, Part 1, Ch. 2, §7].) Here A, K, M are 
the subgroups of G corresponding to the Lie algebras a, €, m respectively. 

Recall our element z e f). Let z — Zi -\- Za he os above. Then i — — m^* © a. Let 
c(m^') denote the center of m^'. Then c(m^*) C i and c is the direct sum of c(m^') and 
a. Since z is not annihilated by any real or complex roots, the intersection of f) with the 
derived part of m^' is elliptic. Therefore (25) applies to M/M^' and c'''^^(m^') in the place 
of G/Z and c^^». 

Furthermore, if x e c''^^ then Xi e c''^^(m^') and det(ad x)n 7^ 0. Hence, we see from 
(27) that for x e c''^^. 



M^)\ ij{g.x)d{gZ) = \7Tm/mH{xi)\ ^(m.x) (i(mM^'). (28) 

Jg/z Jm/mh 

Also, it is easy to see that for any N >0 there are Cjv, N' such that if |V'(y) | < (1 + |y|)~^) 
y e Q, then 

IV^(y)l<CAr(i + |y|)-^' {ye I). (29) 
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Hence for x = Xi + Xa G c*"^^ we have 

\ilj{m.x)\ = |'0(m.Xt + a^n)! < Civ(l + \m.Xi + Xa\)~^' 

< C'j,{l + |m.Xt|)-^'/'(l + |x„|)-^'/2 . (30) 

Thus, (28), (30) and (25) imply our theorem. □ 

Corollary 4. Theorem 3 eoctends to the case where G is a real reductive Lie group with 
finitely many connected components. 

Proof. Suppose first that G is connected. The Lie algebra of G decomposes as g = 3(0)©g" 
where 3(g) is the center of q and q" = [q.q] is semisimple. Write x = ^^(g) + x" for the 
corresponding decomposition of x G 0. The center Z(G) of G has Lie algebra 3(0), and 
G/Z(G) = Ad(G) is a connected semisimple Lie group with Lie algebra q". A Cartan 
subalgebra in q is of the form f) = 3(g) © f)" where f)" is a Cartan subalgebra of q". The 
roots of (P)c, flc) coincide with those of (t)'^, g^) and vanish on 3(g). Let z = 2:^(3) + z" G f). 
If Z is the centralizer of z in G, then Z/Z(G) is the centralizer of z" in G/Z(G). The Lie 
algebra of Z/Z(G) is 3' = {q"Y" ■ Write 3' = c' © 3", where c' and 3" are respectively the 
center and the derived Lie algebra of 3'. Then the Lie algebra 3 = of Z decomposes as 
3 = 3(0) © 3' = c © 3" where c = 3(g) © c' is the center of 3. For x = ^^(g) + x" E c we 
have 7T^/^{x) = 7rg///y(,x"). In particular, the regularity of x depends only on the regularity 
of its semisimple part x". Furthermore, g.x = ^^(g) + g.x". Thus the inequality stated in 
Theorem 3 for G follows from the corresponding inequahty for G/Z(G). 

Suppose now that G has finitely many connected components. Let Gq be the connected 
component of the identity. Then Gq is an open normal subgroup of G. Let Z and Z° be 
respectively the centralizers of 2; G f) in G and in Gq. Then Z*^ = ZnGg is normal in Z and 
GqZ is normal in G. Let tt : G — > G/Z be the canonical projection. Then t : Go/Z° — > G/Z 
given by i(5'Z°) = gZ is an open embedding so that 7r^^i(Go/Z°) = GqZ. In particular, we 
can normalize the invariant measure on Go/Z° = t(Go/Z*^) so that it agrees with the one 
induced by G/Z. Moreover, if xiGqZ, . . . ,a;nGoZ, with Xj G G, are distinct elements of 
G/GqZ, then G/Z = [J^^^SjL{GQ/Z^) (disjoint union). Hence, if / is a sufficiently regular 
measurable function on g, then 



L 



n » 

f{g.x)d{dZ) = Y, / /(s,-5-^)%Z°)- 



G/Z -^^ JGo/7P 

If II • II denotes the operator norm in some finite dimensional representation of g on a 
Hilbert space, then for all j there is a constant Cj > so that for all G G we have 
\\g.x\\ = \\sJ^Sjg.x\\ < Cj\\sjg.x\\ . Since all norms on g are equivalent, we obtain for a 
suitable constant C > 0, 



[ {l + \g.x\)-''d{gZ) = 7r,/,{x)j2 [ {l + ls.g.xD-'^digZ) 

Jg/Z ~l ^Go/ZO 

< C7r,/,{x) [ {l + \g.x\)-'^d{gZ'). 
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Hence the inequality of Theorem 3 for G follows from the corresponding inequality for 
Go. □ 

We keep the notation introduced at the beginning of this section. Recall the derived 
Lie algebra 3" = [3,3] of 3. Assume 3" is isomorphic to a symplectic Lie algebra 5p2n(l^) or 
sp2n(C). Let Z" be the commutator subgroup of Z. Then Z" is a Lie subgroup of Z with 
the Lie algebra 3". Let O" C 3" be a non-zero minimal nilpotent orbit. Fix an element 
n e O" and let Z" C Z be the centralizer of n. Then 

Z/Z" = Z'VZ"" = O". 

Since there is an invariant measure on O" we see that the group Z" is unimodular. Let 
d{gZ'^) denote an invariant measure on the quotient G/Z". 

Proposition 5. Suppose the Cartan subalgebra 1) C. g is fundamental. Then, under the 
above assumptions, for every M > there is N > such that 



sup(l + |a:|n7rg/3(x)| / (1 + %Z 



') < 00. 



Proof. The proof is entirely analogous to the proof of Theorem 3. For reader's convenience, 
we repeat it with necessary modifications. 

We consider first the case when 3" is isomorphic to 5p2n(M). The Cartan subalgebra 
() n 3" is elliptic. Let 6 be the Lie algebra of the maximal compact subgroup K C G 
containing the compact Cartan subgroup H C G. Then n^m/t) = 7rfn3"/f)n3". Hence, a 
result of Rossmann, [Ros90, Corollary 5.4, page 283], shows that there is a non-zero 
constant C e C such that for all / e C^{^) and all x e c. 



d{nn,/^) (^r./^ix + y) [ fix + g".y) d{g"}i)) 
^ C [ f{x + g".n)d{g"Z^), 

Jz/Z" 



(31) 



where j/ G f) fl 3". (We shall verify the assumptions of [Ros90, Corollary 5.4, page 283] 
in Appendix A.) Here the differential operator ^(Trtnj/f,) acts with respect to the variable 



yei)nf. 

Let i/j e C^id). Then (31) shows that for x e c''^^, 

^(TTfna/f)) (7r3/[,(a; + y) / ^(^.(x + |/)) rf(^H) 



G/H 



(32) 

y=0 



[ ^(TT^nj/f,) K/f,(a; + 2/) / i>{g.{x + g".y))d{g"li) 

Jg/Z \ Jz/B. 

C f ij{g.{x + n))d{gZ^). 

7g/Z" 



d{gZ) 

y=0 
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Let f] e P(l))'^(*n3,f,) be viewed differential operator of degree zero, i.e. multiplication 
by 77. Then 

{9{nni/t,)v) lo = v{Q)d{nni/t,): (33) 
where the left hand side is the local expression at zero of the composition of the two 
differential operators and ri{0) G C is the value of 77 at G P). Indeed, there is a unique 
C e ^(1)) such that the left hand side of (33) is equal to d{C). Since Titm/t) is W{tn}, [))-skew 
invariant, so is (. Lemma 10 in [Har57a, page 100] implies that there is ^ G 'p([))^(^'^3,W 
such that C = ^TT^nj/f)- But, by the definition of the degree of ( is les or equal to the 
degree of Tr^n^/i). Hence ^ is a constant. 

Let 7]^ be the sum of the homogeneous components of positive degrees of rj, so that 
7] = r]{0) + r]+. Then <9(7rtn3/(,)(77+vrtn3/i,)(0) = 0. Hence, 

Therefore, 

which shows that ^ = 7^(0), and (33) follows. 

We apply (33) with r) — TTg/^, which is W{t fl 3, ()) -invariant. Then for x G c^'^^ and 

(<9(7rtn3/f,)7rg/3(x + y))\y^„ = {-T^s/^ix + y)d{nini/tj)) j^^g • (34) 
Since TTg/f, = TTg/jTr^/f,, (32) and (34) show that for ip G C^(0) and x G c'"^^. 



9{nni/t,) 7rg/(,(x + 7/) / ^lj{g.{x + y))d{g}i) 

Jg/h 



(35) 



= C7r,/,ix) [ ij{g.{x + n))d{gZ^). 

Jg/Z" 

Since C 7^ 0, (35) together with a theorem of Harish-Chandra, [Har57b, Theorem 2, page 
207 and Lemma 25, page 232] imply that for any M > there is a seminorm u on S{q) 
such that for all V e C^id) and all x G c''^^ 



M 



'G/Z^ 

Notice that for each x G c^^^ the formula 



7^8/3(2;) / '^{g.{x + n))d{gT 



< v{tp)- (36) 



{l + \x\r\7^,„{x)\ / ^(5.(a; + n))d(5Z") 

7g/Z" 

defines a positive Borel measure on q. Hence, [Var77, Lemma 8, page 37] shows that there 
are finite non-negative constants C, N such that 

{l + \x\f'\T;,/,{x)\ f {l + \g.{x + n)\)-''d{gZ^)<C (x G O. (37) 
^g/z» 

This ends the proof for the case when 3" is isomorphic to sp2„ 
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Suppose now that 3" is isomorphic to sp2„(C). Let O C 3" be the unique non-zero 
minimal nilpotent orbit and let /iq be a (non-zero) invariant measure on O. Let f ^ f 
denote the Fourier transform on 3" defined with respect to the Killing form. Then fiQ 
may be computed as in the real case; see for instance [PrzOO, proof of Proposition 9.3]. 
In particular the restriction of fiQ to i)" = t) H 3" is the reciprocal of the product of the 
long roots of [)" in 3^. Hence, 7rj"/t)"/''0 is the product of the short roots. Therefore, with 
an appropriate normalization of all the measures involved, for a test function e -S'(3") 

ip djiQ = I fio{x)tp{x) dx (38) 
1 



\W{Z" H")| 7^,,/^°^^^^^"/""^^^^ J^^^^^^^'i'i9-x)d{gB.")j dx 

C ^(/ioTrj'Vf,") (7fy7f,//(|/) / ip{g.y)d{gR" 

\ Jz"/ii" 



'Z"/H" / y=0 

where C is a constant. The last equality follows from the fact that for complex semisimple 
groups the Harish-Chandra orbital integral commutes with Fourier transform, see [Har75, 
Lemma 35.1, page 198]. The proof for the real symplectic case carries over, with (31) 
replaced by (38). □ 

4. A Weyl Harish-Chandra formula on the odd part of an ordinary 

CLASSICAL Lie superalgebra 

In this section we restrict ourselves to real reductive dual pairs (G, G') which are ir- 
reducible, that is no nontrivial direct sum decomposition of the symplectic space W is 
simultaneously preserved by G and G'. Irreducible reductive dual pairs have been classi- 
fied by Howe [How 79] and are of two types: 

Type I: (O^,,, Sp^jM)) , (0,(C), Sp^JC)) , (U,,„ U.,,) , (0^„ Sp,,,) 
Type H: (GL,(D), GL^(D)) (D e {R, C, H}) 

(In the type I case, one should add to each (G, G') the pair (G', G), if not already listed.) 

According to [Prz06, section 2], we may view an irreducible reductive dual pair (G, G') 
acting on the symplectic space W as a supergroup (S.s). Here S is a Lie group isomorphic 
to the direct product G x G', and 5 = % © Sj is a Lie superalgebra with even part % 
equal to the Lie algebra of S and odd part Sj equal to W. The Lie superalgebra s can be 
reahzed as a subalgebra of the Lie superalgebra End(V) of the endomorphisms of a finite 
dimensional (Z/2Z)-graded vector space V = Vq © V j over D = M, C or H. The vector 
space V = Vq © Vj is called the defining module for (S,5). 

We denote by [■, ■] the superalgebra Lie bracket on 5. Its restriction to x,y E Sj coincides 
with the anticommutator 

{x, y}^xy + yx e % 

of the endomorphisms x and y of V. If x e % and y e Sq, (a e Z/2Z) is homogeneous, 
then [x,y] — xy — yx & Sa is the usual commutator in End(V). The adjoint action of S 
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on Sq, Sj and s is given by conjugation by elements of S. In the following, S-orbits always 
mean adjoint orbits. The S-orbit of x is denoted by S.x. 

Recall that there is a non- degenerate, S- invariant, bilinear form (•, •) on s such that 
(■,-)|j5_xs^ is symmetric, (■,-)|g_x<i- is skew-symmetric, and Sg is orthogonal to Sj with 
respect to (■,■). Furthermore, there is a unique automorphism 6' of 5 such that the 
restriction of 9 to 5q is a Cartan involution and the restriction of 9 to Sj is a negative 
compatible complex structure, [Prz06, Theorem 2.19]. In particular 

-{9;.) (39) 

is a non-degenerate positive definite symmetric bilinear form on s. 

An element x & s is called semisimple (resp., nilpotent) if x is semisimple (resp., 
nilpotent) as an endomorphism of V. Thus x is nilpotent if there is A; G N so that x^ = 0. 
If D = R or C, then x is semisimple provided it is diagonalizable over C; if © = H, we 
consider V as a vector space over C by identifying H as a 2-dimensional right vector space 
over C (see e.g. [Kna02, p. 61]). 

Every element x G End(V) admits a Jordan decomposition x = Xg+Xn with Xg semisim- 
ple, Xn nilpotent and so that XgXn = XnXs] see e.g. [Bou03, Ch. VII, §5, no. 9]. If y G Sj, 
then Xg G Sj and Xn G Sj. Moreover, an element y E Sj anti-commutes with x if and only 
if it anti-commutes with Xs and x„. See [Prz06, Theorem 4.1]. 

Suppose is a semisimple Lie algebra consisting of endomorphisms of a vector space M. 
It is a classical property (see e.g. [Bou72, §6, n^3]) that a; G is nilpotent (or semisimple) 
if and only if ad x is nilpotent (or semisimple) in End(0). The elements of have a similar 
property. 

Lemma 6. Let x G Sj. Then x is nilpotent (resp. semisimple) if and only if adx is a 
nilpotent (resp. semisimple) endomorphism of s. 

Proof. If X is nilpotent, there is A: G N so that x'' = 0. We prove that (ada;)^*^~^ = 0. 
Indeed, [x, y] = xy i: yx for y E Sj^ (/3 G Z/2Z). Hence, for m G N, 

{adxr{y)= J2 Ca,bx''yx\ 

a,b>0 

a+b=m 

where Ca,b G Z. If m = 2k — 1 and b < k, then a = 2k — 1 — b>k — 1 and = 0. So 
{adxy^~^{y) = 0. Thus adx is nilpotent in End(s). 

If X is semisimple, there is a basis B — {ei, . . . , e„} of Vc and elements Ai, . . . , A„ G C 
so that x{ej) — XjCj for all j = 1, ... , n. Let Eij denote the n x n- matrix with all entries 
but the ij-th which is equal to 1. Then x admits matrix representation x — Yl^=s ^sEs,s 
with respect to B. Hence 

n 

adx{Eij) = J2 K[Es,s, Eij] = (A^ - \j)Eij . 

j=s 

Indeed Es,s £ End(V)o, so [£'s,s, Eij] is the usual commutator of matrices. It follows that 
a,dx is diagonahzable in End(Vc), hence semisimple in End(V). 
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Suppose now that a; G fix is semisimple. Let x = Xg + Xn he its Jordan decomposition 
with Xs e Sj semisimple and x„ e Sj nilpotent. Then adx = adx^ + adx„. By the above, 
adxg is semisimple and ad,T„ is nilpotent. Then ada;„ = ad a; — adxg is at the same 
time semisimple and nilpotent, so ada;„ = 0. This means that ,t„ anticommutes with all 
elements of i-^- Xn G ^^Sj. By [Prz06, Lemma 13.0'], ^15^ = 0. Thus Xn = 0, and 
X — Xs is semisimple. 

The case where ad a; is nilpotent can be proven in a similar way. In fact, one does 
not need to invoke [Prz06, Lemma 13.0']. In fact, one now gets adxg = 0, i.e. Xg 
anticommutes with all elements of Sj. In particular, it anticommutes with itself. So 
xl = l/2{xs,Xs} = 0. Thus Xg is at the same time nilpotent and semisimple, i.e. Xg = 
and nilpotent. □ 

As in the classical Lie algebra case, the property of being semisimple or nilpotent can 
be stated in terms of adjoint orbits. 

Lemma 7. Let x e Sj. Then x is semisimple if and only if the orbit S.x is closed. It is 
nilpotent if and only ifO& S.x. 

Proof. The first part is [Prz06, Theorem 4.3]. To prove the second, suppose first that 
X G 3y is nilpotent. Then — X g G S .X by [Prz06, Theorem 4.2]. Conversely, suppose 
G S.x. Let Qn E S so that = hm„_^oo fl'n-a^- Then = hm„_^oo(fi'ri-a;)^ = lim„^oo fl'n-a^^- 
Hence G S.x^. Since x'^ = l/2{a;,a;} G %, the classical property of nilpotent adjoint 
orbits in reductive Lie algebras, e.g. [Var77, Lemma 4], proves that x^ is nilpotent. Thus 
X is nilpotent. □ 

Lemma 7 shows that our definitions of semisimple and nilpotent elements agree with 
those in [DK85]. Following [DK85], we say that a semisimple element x G Sj is regular if 
nonzero and dim(S'.x) > dim{S.y) for all semisimple y G Sj. 

Let a; G 5y be fixed. The anticommutant of x in Sj is 

= {1/ G 5t : {a;, = 0} . 
The double anticommutant of x in Sj is 

'^Bj= fl ^sj. (40) 

Notice that x G ""^^Sx ^"^^ ^^T — ^^^^St- We define a Cartan subspace [}j of 5^ as the 
double anticommutant of a regular semisimple element x G Sj. We denote by the 
set of regular elements in The Weyl group W(S, is the quotient of the stabilizer of 
[)y in S by the subgroup which acts trivially on f)^. 

The following lemma relates our definition of Cartan subspace in Sj with the one given 
in [DK85]. 

Lemma 8. Let x e Sj and let Cx — {y & Sj : [$q, y] C [sq, a;]} . Then c^ — "^^Sj- 

Proof. By definition, y G ""^Sj if and only if {z, y} — for all z G ^Sj, i.e. if and only 
if ^Sj C ffij. By [Prz06, Lemma 3.5], for any y G Sj, we have [%, y] = (^-Sx)"*", where 
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_L denotes the orthogonal with respect to (•, •)|s_xsj- Hence ^Sj C if and only if 
[so,y]Q[so,x]. □ 

Notice that the above definition of regular semisimple elements in Sj is slightly different 
from the one given in [Prz06]. Correspondingly, our notion of Cartan subspaces is slightly 
less restrictive, and allows us to treat also the cases (1) and (2). The study of the 
density properties of semisimple elements given in Proposition 6.6 of that paper needs 
to modified. It is carried over in this section and the main density result is Theorem 19 
below. Observe however that outside the two cases (1) and (2), all our definitions agree 
with those in [PrzOG]. Most of the results in [Prz06, section 6] carry over for all dual 
pairs. Exceptions are the second equality in Lemma 6.9(b), and Proposition 6.10(b) and 
(c). Also, in the proof of [Prz06, Corollary 6.21] the reference to (4.4) and (6.10.b) has to 
be replaced by (6.11.c). 

The next proposition collects the results from [Prz06] we shall need. As in [Prz06], 
we exclude from our considerations the dual pair (Oi, Sp2„) over R or C, as Sj does not 
contain any nonzero semisimple element in this case. (Indeed, for every ^ x G 5j, we 
have 6 Sq = 0, so X is nilpotent.) 

Proposition 9. Suppose (S,5) corresponds to a dual pair different from (Oi,Sp2„) overM. 
or C. Then: 

(a) There are finitely many S-conjugacy classes of Cartan subspaces in Sj. 

(b) Any two elements of a Cartan subspace i)j C Sj commute as endomorphisms of\/. 

(c) Every element of fij is semisimple and every semisimple element of Sj is contained 
in a Cartan subspace. 

(d) The Weyl group W(S, P)y) is a finite group of linear automorphisms of\)j. 

Proof. (Sketch) Let x & Sj and let W C V be a (Z/2Z)-graded subspace preserved by x 
(i.e. x(}N) C W). The element x, or the pair (x,W), is said to be decomposable if there 
are two proper non-zero (Z/2Z)-graded subspaces Wi, W2 of W which are preserved by 
X and so that W = Wi © W2 (orthogonal direct sum if (S,s) is of type I). Otherwise, 
(a;, W) is said to be indecomposable. For every x e Sy, the pair (x, V) is the direct 
sum of indecomposable elements, see [Prz06, Theorem 5.1]. The complete list (up to 
conjugation by an element of S) of Cartan subspaces \)j C sj and Weyl groups W(S, ^j) 
such that t)Y contains a regular semisimple indecomposable element is given in [Prz06, 
Proposition 6.2]. In the general Cartan subspace C sj induces a direct sum 

decomposition of V into indecomposable pairs {xj,\/^) with Xj e fiy. Then [)j will be 
direct sum of Cartan subspaces in each of these indecomposable pieces. They can be read 
off from the above lists. Likewise, the Weyl group W(S, fij) can be reconstructed out of 
those of the indecomposable pieces; see [Prz06, page following the proof of Proposition 
6.2]. The description of each Cartan subspace as double anticommutant of a regular 
semisimple element is given in [Prz06, pp. 487-498] , based on the classification of [Prz06, 
Proposition 6.2]. From the explicit expressions obtained from this case-by-case analysis, 
one also deduces the remaining statements. □ 

The special nature of the dual pairs (1) and (2) is explained in the next proposition. 
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Let V = Vq © Vj be the defining module of (S, 5). For a fixed semisimple x e 57 we can 
decompose V = V° © V"*", where V° = ker(a;) and V+ = xV are (Z/2Z)-graded subspaces. 
Wc shall denote by (S(V+), 5(V+)) and (S(V°),s(V°)) be the supergroups obtained by 
restricting (S,s) to V+ and V°. 

Proposition 10. Let V = V° © V"*" and x E s- regular semisimple be as above. Then 
Sy(V°) 7^ {0} if and only if {S,s) corresponds to one of the dual pairs in (1) or (2). In 
cases (1) and (2), we have drnxj^ Sjiy'^) = 2n — (p + g — 1). 

Proof. Suppose first that (G,G') is not (0^,^, Sp2„(M)) or (Op(C), Sp2„(C)). We shall 
prove that 5i(V°) = 0. 

We begin by proving that St(V°) does not contain any nonzero semisimple element. 
For this, it suffices to prove that Sy(V°) does not contain any nonzero semisimple y so 
that (y, V°) is indecomposable. (This notion has been introduced in the proof of the 
Proposition 9 above.) Indeed, by [Prz06, Theorem 5.1], each pair (2/, V°) with ^ y 
semisimple is a direct sum of nonzero indecomposable elements, which are themselves 
nonzero semisimple elements in 5y(V°). 

We argue by contradiction. Suppose there is ^ y & St(V°) semisimple so that (|/, V°) 
is indecomposable. Then, up to similarity, y is a non-zero element of the Cartan subspaces 
listed in [Prz06, Proposition 6.2]. A case-by-case inspection shows that there exists an 
element in 5q(V°) not commuting with y. Hence 

dimSo(V°)^ < dimSo(V°) . (41) 

Notice that C = 5o(VO) © So(V+)^' and So(VO) C 52. Hence 

4 = ® (%(V+)^')^ . (42) 

As y|v+ = 0, we have So(V+) C s^. Therefore 

4+^ = 4 n 4 = %(v°)^ © {Boiy^fr ■ (43) 

As x\\/o = 0, the elements x and y commute. So a; + y is semisimple. From (41), (42) 
and (43) , we get dims^"*"^ < dims^. Using the relation dim 5^ = dimSQ — dim[5Q, z], we 
therefore obtain that dim[a;,%] < dim[x -|- 1/,%], contradicting that x is regular. Thus 
5y(V°) cannot contain any nonzero semisimple element. 

Now (S(V°),5(V°)) is a supergroup coming from a dual pair. If 5j(V°) 7^ 0, the only 
case where 5j{\/^) contains no nonzero semisimple elements is when S(V'^) = Oi x Sp2m. 
over M or C for some m e N. This means that (S,s) corresponds to (Op,g, Sp2„(M)) or 
(Op(C), Sp2„(C)). This case was excluded. Hence Sj{y^) = 0. 

We now turn to the case where (G, G') = {Op^q, Sp2„(M)) or (Op(C), Sp2„(C)). 

Notice that the proof above also shows that if 5-(V°) 7^ then S(V'^) = Oi x Sp2m 
over D = M or C, for some 1 < m < n. Thus 5x(V°) = D^*". (The precise value of 
m as a function of the given p, q and n will be computed at the end of the proof.) 
Moreover dimD(V° fl Vq) = 1 and dimD(V° fl Vj) = 2m. Since a;|v+ is injective, we have 
x(VonV+) C VxnV+ and x(Vi nV+) C VonV+. Hence dimD(VonV+) = dimD(VinV+). 
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It follows that 

dimo(VT n V+) = dimD(Vo n V+) = dimo Vq - dimD(V(y n V°) = p + g - 1 . (44) 
(Here and in the following, we set g = when © = C.) Therefore 

2n = diniB Vy > dimD(Vx n V+) = p + g - 1 . 

This proves that St(V°) = if p + g > 2n. 

Suppose now p + q < 2n. Notice that ker(a;^) = ker(a;) as x is semisimple. Hence 
ker(a;^|vQ) = ker(a;|vy) = Vq fl V°. The element x is regular and x'^\\/- belongs to a Cartan 
subalgebra of 5o(Vq), the Lie algebra of G = Op,^ or Op(C). So dim (ker(a;^|vQ)) = 
dim(VQ n V^) = 1 if and only if p + g is odd. 

Finally, if p + g < 2n and p + g is odd, then 

dimD(V° n Vi) = dimn - dimD(V+ n V^) = 2n - (p + g - 1) > . (45) 

Thus n Vt ^ and n 0, which yields 5t(V0) 0. 

Formula (45) also implies that the constant 2m obtained before as dimD(V° n Vj) is 
equal to 2n — (p + g — 1). Thus dimD5x(V°) = 2n — (p + g — 1). □ 

Recall that the rank of a reductive Lie group G, denoted by rankG, is the complex 
dimension of a Cartan subalgebra of Qc, the complexification of the Lie algebra of G. 

Corollary 11. If x E l)i is regular, then — "^Sj. Moreover, 

dim[So, x] + dim l)j + dimSY(V'^) = dimsj . (46) 

All Cartan suhspaces of Sj have the same dimension. It is equal to min{rankG, rankG'} 
i/(G,G') is the dual pair associated with (S,s). 

Proof. The first part is an immediate consequence of Lemma 8. Indeed ^\va.S.x — 
dim[sQ, x]. Hence, if f)j = %y and a; G f)x is regular, then [Sg, x] = [%, y] and = Cy. 
For the dimension formula (46), we have f)^ = Sy(V+)^ by [Prz06, Proposition 6.10(a)]. 
Since ker(a;|v+) = 0, [Prz06, Theorem 4.4(a)] gives dim5Y(V"'")^ = dim'^(sY(V+)). More- 
over, = -Sy(V°) © '^(sy(V+)). By [Prz06, Theorem 3.5], dimfsg, x] = dimSY — dim^Sp 
This proves (46). The formula for the dimension of the Cartan subspaces is from [Prz06, 
Theorem 4.4(c)]. □ 

Remark. Because of Lemmas 7 and 8, the fact that the elements of a Cartan subspace are 
semisimple can also be deduced from [DK85, Lemma 2.1]. Since [DK85] considers actions 
on complex vector spaces, this lemma can be directly applied when Sj admits a complex 
structure commuting with the group action o/S. This is equivalent to the fact that \/ is a 
vector space over C. (Recall that we are considering V as a vector space over CifD — M.) 

When V is a vector space overM.. it is necessary to consider the complexification (Sc,5c) 
of the supergroup (S,5). The list of irreducible dual pairs gives the following two possibil- 
ities: 

S = GL„(R) X GL„(R) Sc = GL„(C) x GL„(C) 
S = Op,, X Sp2„(R) Sc = Op+,(C) X Sp^jC) 
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Notice that ifM is the defining module of {S,s), then the defining module of (Sc,5c) is 
Vc- Hence, for an element x e Sj, to he semisimple means that x is semisimple as an 
element of {s-)c = (sc)t ^ End(Vc). 

Let i)j = ""^Sj be a Cartan subspace in Sj. Then i)j C ''^^'^h(5c)i. can deduce from 
[DK85] that i)j consists of semisimple elements if we prove that '^*-^'^'*t(sc)t ^■s « Cartan 
subspace in {sc)j, i-e. that x is regular in {sc)t- Since Sc is a complexification of the 
Lie group S, the last part of Corollary 11 shows that the (real) dimension of a Cartan 
subspace in (5c)t ^■^ twice the dimension of a Cartan subspace of Sj. Apply formula 
(46) to both (S,s) and (Sc,Sc). Corollary 11 also shows that dims-(V°) is the same for 
all spaces V° = ker(a;) with x regular semisimple in Sj. This constant doubles in the 
complexification. It follows that a semisimple element y e (sc) i is regular if and only if 
dim[{sc)o,y] = 2 dim[5Q, x], where x is the fixed regular semisimple element of Sj. Since 
(-Sc)o = (So)c; it follows that x is regular in (sc)t- 

Even if this argument allows us to deduce the semisimplicity of elements in the Cartan 
subspaces from the general results o/[DK85]; it uses many properties [Prz06]. So it still 
(strongly) depends of the case-by-case analysis made in [Prz06]. 

Recall the defining module V = Vq © Vj for (S,5). Let [)y C be a Cartan subspace, 
and let V'' C V be the intersection of the kernels of all the elements on fjj. Equivalently, 
V° = ker(a;) if f)x = ""^Sj. Define 

^t = 5t(V°)®^x. (47) 

By Corollary 11, i)j ^ i)j ii and only if (S,5) corresponds to a dual pair in (1) or (2). 

We say that an element x & Sj is almost semisimple if it is conjugate to an element of 
l)i for a Cartan subspace fij C sj. 

Our next main result (Theorem 19) shows that the almost semisimple elements are 
dense in Sj. Notice that semisimple elements are almost semisimple by Proposition 9. We 
now study how nilpotent elements can be approximated by almost semisimple elements. 
For this, we first consider some suitable root spaces decompositions in s. 

Let f)Y C 5- be a Cartan subspace and let f)| C Sq be the linear span of all the an- 
ticommutants of the elements of Then 1)?^ is generated by the pairwise commuting 
semisimple elements x'^ — l/2{a;,a;} with x e Moreover, i)j is contained in some 
Cartan subalgebra of Sq. We may assume that f)^ is ^-stable. Define 

a = {xe^^:9x = -x}. (48) 

For each a; e o the operator ad a; e End(s) is symmetric with respect to the form (39). 
Hence, the Spectral Theorem imphes that 5 decomposes into an orthogonal direct sum of 
the eigenspaces for the action of a: 

s = ^Sa, where Sx = {y G s : [x, y] = \{x)y for all x G a}. (49) 

Each 5a is (Z/2Z)-graded. Let I = 5o denote the zero eigenspace, i.e. the centralizer of a 
in 5. The non-zero eigenfunctionals A e o* are called roots (of a in s). Thus for a root A 
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we have a non-zero y G s such that 

[x,y] = X{x)y (xea). (50) 

Notice also that if y G s is a root vector corresponding to A € a*, then 6y is a root vector 
corresponding to —A. Furthermore, [sa,^/^] Sx+h- 

Let a^^^ C a be the subset where no roots vanish. Fix an element Xq G a^^^. We say 
that a root A is positive, written A > 0, if A(a:o) > 0. Set 

A>0 

Then n C 5 is a Lie sub-superalgebra and 

A>0 

Hence (49) may be rewritten as 

5=l®n®6in, (51) 
Notice that S\ -L ii X > and /i > 0. Indeed, if y G Sa and z E then 

Kxo){y,z) = {[xo,y],z) = -{y, [xo,z],) = -ii(xo){y, z), 

which is possible if and only if {y,z) — 0. Thus the restriction of the form ( •, •) to n is 
zero. Similarly, we check that I _L n and I ± ^n. 

On the other hand, the Spectral Theorem implies that the restriction of the form (39) 
to n is non-degenerate. Hence the form (-, ■) provides a non-degenerate pairing between n 
and ^n. Thus the restriction of the form (•, •) to © n is non-degenerate. Similarly, the 
restriction of the form (•, •) to 1 is non-degenerate. Finally we notice that n is the radical 
of the form (■, ■) restricted to i + n. 

From the structure of the Cartan subspaces [Prz06, Theorems 6.2, 5.1 and 4.4], we see 
that the following lemma holds. 

Lemma 12. The Lie superalgebra I decomposes as 

n n+m 
i=l i=n+l 

where {lo)j — in the type II case and Iq is of the same type as $ in the type I case. 
For 1 < i < n, li is isomorphic to {Qii{3) , Qii{3)) as a dual pair, where D = M, C or 
H. If 3 ^ M. then m = 0. // D = M then, for n + l<i<n + m, li is isomorphic to 
(0t2(lR), 0l2(IR)) as a dual pair. In all cases f)^n {Iq)q (which might be zero) is elliptic and 

( Xl"=i" (Qo) fundamental. 
Corollary 13. With the notation of (51) we have dimn-Q = dimxij. 
Proof. We see from (51) that it suffices to check that 

dim Sq — dim Iq — dim Sj — dim Ij. (52) 
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For this we use the classification of the irreducible real reductive dual pairs, see e.g. 
[How89, p. 548]. Let V = Vq© Vj be the defining module of the pair, and set d — dimuVo 
and d' — dimn Vj. 

Suppose s is of type II. Then 

dimSQ — dimigD • (d^ + d'^), 

dimsj — dirriM.^ ■ 2dd' , 

dimlo = dimT&Bi-{2n + 8m+(d-d')^), 

dim Ij — dim^ D • (2n + 8m), 

and (52) follows. 

Suppose 3 is of type I. Let i = 1 if D 7^ HI and i = | if D = HI. We may choose 
e, e' e {0, |, l} such that e + e' — l and 

^ fd'id'-i) , did-i) , 
dims^ = dim^ © • ( — — - + d'e' + -^^ — - + de 

dimSj — dim^'O ■ dd\ 

r ^ f id' - 2n - Am)id' - 2n - Am - I) , ^ , 
dim % = dim^ D • f ^ + (d' - 2n - 4m)e' 

(d-2n-4m)(d-2n-4m-l) ^ , , ^ „ \ 

+ ^ - + {d-2n-4:m)e + 2n + 8mj, 

dim I- = rfimjftD • {{d — 2n — 'im){d' — 2n — 4m) + 2n + 8m). 

Hence, (52) follows. □ 

By definition, [ and n are Lie superalgebras and [ normalizes n. Thus, 

[%, y C nj, {xij, Ij} C Uq and [%, ny] C ux, . (53) 

Because of Corollary 13, the vector spaces % and xij are isomorphic. Let T : rij ^ % 
be any fixed isomorphism. Let : Uq ^ nj and : Uj ^ Uq be linear maps depending 
on a variable x. Then T o and oT are endomorphisms of tIq and nj, respectively. 
Define 

det (Ax) uj^^n- = 

dct(ToA,) (54) 
det(fi,)n-^n^ = dct(fi,oT-i). 

Then dct^B^oAx) = det{Bx)nj-^njj'iet{Ax)nj^^nj- Moreover, B^oA^ is an automorphism of 
Uq if and only if A^ and B^ are isomorphisms. This is in turn equivalent to the condition 
that det(A^)n-^„T 7^ and det(5^)n^^no 7^ 0. 

Lemma 14. Suppose x & Ij is such that det(ad x'^)^^ ^ 0. Then the map 

% 9 — >■ exp(2;).x — X e rij (55) 
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is a well defined analytic dijjeomorphism 0/% onto nj. Define the maps 

n^B z ^ [z, x] e xij (56) 
nj3y ^ {y, x} e %. (57) 

The composition (57)o(56) coincides with 

[z, x'^] e %. (58) 

// we identify the vector spaces rig and Uj by fixing a linear isomorphism between them, 
as above (54), then the Jacobian of the map (55) at z is equal to | det(ad a;)n_^„_|; the 
absolute value of the determinant of the map (56). Also, 

I det(ad x\-\ = | det(ad a;)nj->„g det(ad a;)ng^„^|. (59) 

Proof. We see from (53) that the maps (56) and (57) are well defined. Since the adjoint 
action is a derivation and since x"^ — ^{x, x}, the composition (57)o(56) coincides with 
(58). Hence, (59) follows. 
Also, we see from (53) that for z &xIq the element 

exp(^).a; - x = + ]^[z, [z,x]] + ^[^, [z, [z,x]]] + . . . 

belongs to Uj. Thus the map (55) is well defined. 

Let z,y & tIq. As is well known, see e.g. [Ros02, §1.2, p. 15], 

/ \ / \ T I ~ exp(— ad z) , . , , 

exp(— 2;) cxp(2; + = i H y + higher order terms m y. 

ad z 

Hence, the derivative of the map (55) at z coincides with the following map 

/ — exp(— ad z 



n-^3y ^ exp(2;). 



ad z 



y ,x 



e nj. (60) 



Since the map (56) is injective and since z is nilpotent, it is easy to see that the map (60) 
is injective and in fact has the determinant equal to ±det(ad a;)„_^„_. Hence, the map 
(55) is everywhere locally bijective. 

Suppose exp(^). a; = exp(^').a;. Then exp(^).a;^ = {exp{z).xy = (exp(2;').a;)^ = exp{z').x'^. 
Hence, by [Har57b, page 218], z = z'. Therefore the map (55) is injective. 

We already know that the image of the map (55) is open. If we show that this image 
is also closed then the surjectivity will follow. As in [Har57b, page 218], it suffices to 
show that if exp{z).x — a; is bounded then z is bounded. But if exp{z).x is bounded, then 
exp{z).x'^ — {exp{z).xy is bounded. In turn, this implies that z is bounded, see [Har57b, 
page 218]. □ 

Lemma 15. Any element of rij is nilpotent. Any nilpotent element z & lj + nj is of the 
form z — zi + Zn, where both zi e Ij and z^ G rij are nilpotent. 

Proof. Since rij C X]a>o-^^ [sx,Sfj\ C Sx+fj,, the endomorphism adx is nilpotent for 
every x e xij. By Lemma 6, x is nilpotent. 
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Suppose now that z = zi + Zn is nilpotent, and zi G Ij and Zn e rij. By Lemma 6, adz 
is a nilpotent endomorphism of End(5). Let k E N so that (adz)'' — 0. Let A denote the 
set of positive roots. Then z E I + n = X]AeAu{o}-^A and z^ E n — X^aga-^a- Let // be 
either a root or /i — 0. For a, 6 e N we have: 

(adzn)"(5^) Q X] 

Ae/i+aA 

(adz)''(5^) C ^ 5a, 

ASAt+A 

{adzfiadznTis,,) C J] Sa, 

Ae/x+A+aA 

where we have set aA = {Ai + ■ ■ ■ + Xa : Xj E A} and Sa = if A 7^ is not a root. Since 
the set of roots is finite, there is /i e N such that for every root /i and all integers a > h 
the set fjL + a A does not contain any root. For p e N we have 

{adz(Y = (adz - adznY = J] ±(adz)''i(adz„)"i . . . (adz)^^(adz„)"% 

where the summation is over positive integers aj, bj, 1 < i < s, with Ylj=ii^j + ^j) ~ P- 
Fix /X and x E s^, and look at 

z = {adzf'{adznT' . . . {ad zf' {ad z^T' (x) . 

Then z E X^Ae^+A+aA -^m where a = 0,1 + ■ ■ ■ + a^. If s > /i then a > h. So 5a = for all 
X E fi + A + aA, and thus z = 0. Suppose then s < h. U a > h, then, as above, z = 0. 
We can therefore assume a < h. Set b = bi + ■ ■ ■ + bg. Then b=p — a>p — h. If we 
have chosen p sufficiently big (specifically, > h{k + 1)), then there must be an index j so 
that bj > k. Otherwise b < sk, which would give hk > sk > p — h. But if bj > k then 
{adz^^ = 0. We conclude that z = also in this case. 

This proves that ad zi is nilpotent. Because of Lemma 6, zi must be nilpotent too. □ 

Corollary 16. Consider a nilpotent element z E s- with the decomposition z = + 
as in Lemma 15. Suppose (1 ij is a subset containing Z[ in its closure and such that 
det(ad x'^)n- 7^ for all x E Ij. Then z may be approximated by elements of the S-orbits 
passing through Ij. 

Proof. Choose a small element y E Ij such that zi — y E ^. Let x = Z[ — y. We see from 
Lemma 14 that there is an element u Exiq such that 

z^ = exp{u).x — X. 



Hence, 
is small. 



(^n + ^i) - exp{u).x ^ y 



□ 
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Since our proof of the following proposition requires a significant amount of additional 
notation, we shall present it in Appendix B. (In fact what we are missing here is an analog 
of the Jacobson-Morozov Theorem.) 

Proposition 17. For any nilpotent element z E Sj there is a Cartan subspace f)Y Q Sj such 

that, in terms of (51), z belongs to an S-orbit passing through fY + nj. Moreover, Ij D 

Suppose z E I- is nilpotent. Let L C S be the subgroup with the Lie algebra Ig, so that 
(L, I) is the direct product of classical Lie supergroups, as in [Prz06, sec. 2]. Then zi is the 
limit of elements in the L-orbits passing through 

Theorem 18. Every nilpotent elements z E Sj is a limit of almost semisimple elements. 
It is a limit of semisimple elements if and only if {S,s) is not associated with (1) or (2). 

Proof. By Proposition 17, z singles out a Cartan subspace t)x Q 5j. Moreover, if s = 
(©n©^n is the corresponding decomposition (51), then z e S.Fwith z — Z[-\-Zn G lyffitiY 
necessarily nilpotent. Lemma 15 shows that i[ G ij is nilpotent. So, by Proposition 17, 
there are elements In E L and x„ G [)x so that Z[ = lim„^oo In-Xn- By Proposition 17, 
1)Y C i-. The map x — )■ det(ad,x^)n_ is polynomial on li. Since it is not identically zero 
on the set i)j = {x E i)i : dct(ada;^)n_ 7^ 0} is dense in So we can suppose that 
Xn G Corollary 16 with Ij = f)^ therefore assures that J is a limit of elements of the 
(S'-orbits passing through The same property holds thus for z. □ 

Combining Proposition 9, Theorem 18 with Jordan decomposition of an arbitrary el- 
ement of Sj into the sum of its semisimple and nilpotent components, we obtain the 
following theorem. 

Theorem 19. The set of the almost semisimple elements of Sj is dense in Sj. The set of 

the semisimple elements in Sj is dense in Sj if and only if (S,5) is not associated with (1) 
or (2). 

Proof. Let a; = a;^ + a;„ be the Jordan decomposition of an element in a; G Sj. Because 
of Proposition 9(c) and Theorem 18, we can assume that x is neither semisimple nor 
nilpotent. In particular Xs 7^ 0. Since x, Xg and Xn commute in End(V), they all belong 

to sf. As in the proof of Theorem 4.4(b) in [Prz06, section 13], let V = V° © • • • V'^ 
be the isotypic decomposition of V with respect to Xg- Hence V° = ker(xs) and © • • • © 
V*^ = V+ = Xf^J. Isotypic means that each (aj^lvj, V-') decomposes into mutually similar 
indecomposable pieces. Let (S(V^), s(V-')) denote the restriction of (S,s) to V-'. Then 

= s^(V°) © Sy(V')^' © • • • © 5t(V*^)^' . (61) 

Decompose x — xq -\- Xi + ■ ■ ■ + x^ according to (61). If we prove that for j = 1, . . . , A; 
there is a sequence ym,j of semisimple elements so that Xj = limm^ooj/mj, then x = 
limm_^oo xq + yjn,i + • • • + yTn,k wiU be a limit of almost semisimple elements. Notice that 
keT{xs\vi) = 0. Hence, by [Prz06, Theorem 4.4(a)], each 5x(V^)^« is a supergroup with 
corresponding dual pair isomorphic either to (U„, U„) or to (GL„(D), GL„(D)). Here the 
division algebra D may be different from the one over which V is defined. This reduces 
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the proof of our Theorem to the case where x = Xs + Xn belongs to one of these isotypic 
components. A further reduction allows us to select Xg inside non-conjugate Cartan 
subalgebras {)x C s-. Indeed suppose Xg = g.x'^ with £ S and x'g e f)x. Write Xn = Q-x'^- 
Then x'^ is nilpotent and x'^x'^ = x'^x'^. If x'^ + x'^ = limym with i/m semisimple, then 
X = \im g.ym with g.ym semisimple. This case-by-case analysis will be carried out in 
Appendix C. □ 

Recall that the almost semisimple elements in Sj are the elements of the S-orbits of the 
sets 

^T=f)T®5T(V°) 

when f)Y ranges among the (conjugacy classes of) Cartan subspaces in t)Y. Here V° denotes 
the intersection of the kernel of all elements in Recall also that if x G f)- is regular, 
then = kcr(x) and we have the decomposition V = V° © V"*" with V"*" = xV. Let S^t 
denote the centralizer of [)y in S. Then 

8^ = s(V°) X S(V+)''T. (62) 

The group S''^ is reductive and hence unimodular. Therefore there is an invariant measure 
d{gS'^^) on the quotient space S/S*^". 

The title of this section refers to the following theorem. 

Theorem 20. For a continuous compactly supported function (p : Sj ^ C the following 
integration formula holds, 

0(x)(ix = ^ ^ / Itt /^2(x2)| / / <P{g.{y + x))dyd{gS''^)dx^, 

where the summation is over a maximal family of mutually non-conjugate Cartan sub- 
spaces f)- C S-. 

For a fixed x E i}- the integral /^..^vo) <Pig-iy + x)) dy is a function of (? G S, invariant 

under the right translations by elements of S''t. Also, the integral over S/S''t is constant 
on the fibers of the square map x ^ x"^, because the group S) acts transitively on 

these fibers. (See [Prz06, Corollary 6.25].) 

Proof. Notice that for any yo G Sx(V°) \ {0} the S(V°)-orbit through yo is equal to Si(V°) \ 
{0}. Fix any yo G St(V°) \ {0}, then S''t"'"^°, the centralizer of yo + tjj in S, is a unimodular 
group. Furthermore, by (62), S^^+'^t = S(V°)2'o x S(V+)^t. Hence 

/ / (l>{g.{y + x))dyd{gS^T)= f 4>{g.{yo + x)) d{gS^°+^T), (63) 

Also, if the function is supported in the union of the orbits passing through i)j — 
5t(V0) X then 

'^(^) = iw^Q M / / '^(^•(^0 + ^)) digSy°^^~) Jiyo + x) dyo dx, 



26 



M. MCKEE, A. PASQUALE, AND T. PRZEBINDA 



where J is the Jacobian, which may be computed as in the proof of [Prz06, Corollary 
6.21]. In particular, Jiy^ + x) is equal to \'k^_i^'i{x'^^\ times the Jacobian of the square 

map f)Y 9 a; i-^ a:^ e \)\- Since, as we mentioned previously, the integral is constant on the 
fibers of this map, the formula follows. □ 

5. A SEMISIMPLE ORBITAL INTEGRAL ON THE ODD PART OF AN ORDINARY 

CLASSICAL Lie SUPERALGEBRA 

In this section we prove that the integration formula of Theorem 20 extends to Schwartz 
functions on the symplectic space W = 5j. For this, we study whether the orbital integral 
of Theorem 20 is intcgrablc on when = (1 + and > is sufficiently 

large. Here | • | denotes a fixed norm on the real vector space s. Recall that the Cartan 
subspace [)y ^ is called elliptic if and only if all the eigenvalues of the \)\ acting on 
are imaginary. 

Proposition 21. Suppose the Cartan subspace I)y C is elliptic. Then for any constant 
M > there are positive constants C, N such that 

/ . / a + \9-ix + y)\)-''dyd{gS'T)<C{l + \x\)-^' {xei)^')- 

Here the integral over Sjiy^) is omitted if the dual reductive pair is not isomorphic to (1) 
or (2). 

Proof. We may assume that the norm | • | is chosen so that the maximum of when 
X e Sj varies through the unit sphere, is 1. Then 

\x^\<\x\^ {xesj). (64) 

Also, it is clear from the description of the Cartan subspaces in [Prz06] that there is a 
constant 1 < C < oo such that 

\x\^ < C\x^\ {x e (65) 
Thus for any non-negative constants m, N and fl' G S, 

(l+|xr)^(l + |^.(x + y)r)-^ < C^(l + |a;2|)^(l + |^.(a;2 + y2)|)-^ {x e f)T, y e 5t(V°)). 

Notice that since the Cartan subspace \)j C sj is elliptic, the quotient (S/S''t)/(S/S''t) is 
compact and therefore has finite volume. Hence our integral is dominated by the identical 
integral with the x replaced by x^ and the S/S''" replaced by S/S''". 

Now we apply Theorem 3 or Proposition 5 and (31) with G — S, £| — Sq, z — x"^ for any 

X G '^j''^ so that Z = S^T, 1=5-^, [)j C c and tIq/i\^2 = t^s^/^^I- D 

Remark. The inequalities (64) and (65) show that the estimate of Proposition 21 is as 
sharp as the estimate of Theorem 3. In case when Z = H C G is a Cartan subgroup, 
there is a characterization of the regular semisimple orbital integrals on a semisimple Lie 
algebra due to Bouaziz, [Bou94], from which one can see that the estimate of Theorem 3 
is sharp. 
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For AT > define 

t-^ if t>l, 



27 



lN{t) 



Tlien 
and 



1 - N \n{t) if 0<t<l 

[lN{t)f'^ < lN/2{t) {t > 0) 

iNi-s) > /jv(t) (0 < S < t). 



(66) 

(67) 

(68) 



Lemma 22. Suppose that the supergroup (S,s) is such that the corresponding dual pair 
is isomorphic to (GLi(D), GLi(D)), D = M, C or H. Then for each N > 1 there is a 
non-negative, finite constant C such that 

I (1+ |s.x|2)-^d(sS''T) < CIm{\x\) {x e W). 

Proof. We may realize our supergroup in terms of matrices as follows, 

zi 



Z2 



; Zi, G 



Then, by the classification of Cartan subspaces, [Prz06], we may assume that 



z 
ez 



where M = R, C = C and H is a copy of C in H. Moreover, e = ±1 if D = R and e = 1 
in D R. Since, 

-1 



g 
9' 



we see that 



z 
ez 



S^i 



g 

9' 

9 

g 





9'^zg-^ 



9zg 




/-I 



The following formula defines a norm | • | on Sj. 

2 







V ^2 


■i) 



9 e 



\ ^2 ^ J 

where \z\^ — z'z., z Let 

f/(D) = {c, e ; = 1}, uip) = D n Uip). 

Then, as a homogeneous space, 

S/S^T = X (D^/D^) = X (C/(D)/C/(D)). 
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Hence, we may normalize all the measures involved so that 



/ (1+ |s.a;n-^d(sS^T) 

/ / {l + \gzg'-'\' + \g'zg-'\')-''dg'dg 
Jdx Ju{o) 

il + ia' + a-')\z\')-''da/a<-I^{\z\), 



where the last inequahty follows from (D.l). □ 

Lemma 23. Suppose that the supergroup (S,s) is such that the corresponding dual pair 
is isomorphic to (GL2(]R), GL2(M)) and that the centralizer o/ f)| in Sq is a fundamental 
Cartan subalgebra. Then for each N > 8 there is a finite constant C such that 

1 is/sT 

Proof. We realize our supergroup in terms of matrices as follows, 

^={^=(o °/); ^,^'eGL2(M)|, ^T={(° y, z„ Z2 e 

Then, by the classification of Cartan subspaces, [Prz06], we may assume that 

For X E i)j as above, we shall write ^ = Re(2;) and rj — Im(2;). As in the proof of Lemma 
22 we check that 

which is isomorphic to embedded into GL2(M), as a Lie group. Hence, as a homoge- 
neous space, 

S/S^T ^ GL2(R) X (GL2(R)/C^) = GL2(R) x (SL^(R)/S02). 
Define a norm | • | on 5^ by 

2 

= tr{z\zi) + tr{z\z2). 
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we may normalize all the measures involved so that 



/ (1 + |s.a:n-^(i(sS^T) 
= / / {l + \gzg'-'\' + \g'zg-'\ydg'dg 

r p POO 

- {l-r a'\gzg'-'\'-ra-'\g'zg-'\')-''da/adg'dg 

^SL2(K) ^SL2(M) JQ 

< [ I ^M^\3'9-'\\9'^9-'\f'')dg'dg, 

JSL2(R) JSL2(M) 

where the inequality follows from (D.2). Notice that 

\9zg'~^\\g'zg''^\ > \gzg'^g'zg'^ \ = \gz^g'^\. 
Hence, by (68) and (67), 

< /iV/2((|^.VY/^)/iV/2((|^V^'-Y/^). 

Therefore (69) is less or equal to 

I^/,{{\gz'g-'\'/')dg 



2 

^ \^SL2(M) 

We perform the integration (70) in terms of Cartan decomposition (G = KAK): 
/ I^/,{\gz'g-'\'l')dg 

/OO 1 
lM/2mM^^)f + («' + ci-'){Mz^)ff'\{a^ - a-^) da/a 

/OO /*CXD 
lN/2{a\lm{z^)\^''^)a^ daja< J lN/2ia\lm{z'^)\^/^)a^ da/a 

POO 

— |lm(2;^)|~^ / I^i2{a)ada, 
Jo 

where that last integral is finite because ^ > Therefore (69) is less or equal to 



2 
TV 



OO \ 2 



In/2{o)(i da 1 |Im(2; 
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Notice that, by (67), in terms of (71) wc have 

/^/2((2(Rc(^2))2 ^ (^4 ^ a-'){\Mz^)f f") 

< (/^r/4((2(Re(z2))2 + {a' + a-'){\n^{z')ff'')f 

< 7^/4((|^Y/^)/iv/4((2(Re(^2))2 + (a^ + a-^)(Im(^^))^)V4) 

< 7^/4((|^T/^)/iv/4(a|Im(z^)|V^). 

Hence, (71) is less or equal to 

poo 

|Im(^^)|-i/^/4(|^T^') / lNiA{a)ada, 

Jo 

where the integral is finite because N/A > 2. Therefore (69) is less or equal to 



2 

TV 



(73) 



(74) 



Clearly (69) is less or equal to the minimum of (72) and (74). Thus there is a finite 
constant C such that (69) is less or equal to 



C(l + |^|)-^/2|Im(^2 



(75) 



Since |7r5_/f,2(x^)| is proportional to |Im(2;^)p, we see that (75) completes the proof. □ 

Len L, N C S be the Lie subgroups with the Lie algebras % respectively and let 
K C S be the maximal compact subgroup corresponding to the Cartan involution 9. 



Corollary 24. For any non-negative measurable function : — )■ M and a; G f)^ 



reg 



S/S"! 



(I){S.X) d(sS^T) 



7ri^/„2(x^) |det(ad x)„_^„_ 
7r[^/i,2(a;^) |det(ad x\-\^ 



L/LT Jnj JK 



L/LT JnjJK 



(f){k.{l.x + y)))dkdyd{lL^T^) 
(j){k.{l.x + y)))dkdyd{lL^^). 



Proof. Notice that S = KNL and that the Haar measure on S may be written as ds 
dkdndl. Furthermore, Lemma 14 implies that 

/ (f){n.x) dn = I . . r / (t){x + y) dy. 

Jn I det(ad a;)n_^„_| 



We see from (59) that 



det(ad x)n_^„_| 



det(ad x) 

\'Ky^^_{x^)\\dei{&d x%-\ 
I det(ad x)„_^„_| 



k,^/f,2(a^^) det(ad x\.^^.\ 
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This implies the first equahty. Since the maps (56), (57) arc adjoint to each other (see 
[Prz06, Proposition 2.18]) 

I det(ad a;)ft_^n_| = | det(ad x)„-^nj\- 

Hence, by (59), 

I det(ad a;)n-^nol = |det(ad a;^)no|' 
and the last equality follows. □ 

Fix a K-invariant norm | ■ | on the real vector space s. (For example \x\'^ = —{6x,x), 
X e S.) The following theorem guarantees that the Weyl Harish-Chandra integration 
formula of Theorem 20 extends to Schwartz functions on W = Sj. 

Theorem 25. For any constant < M < oo there are constants < N,C < oo such that 
for all X e f)/^^ 

1 JS/ST Aj-(VO) 

n n+m 

< C-|det(ada;%|^-(l + |a:|,„,|)-^-n/M(|a;|i^,|)- \{ (1 + |a;|,^,|)-^. 

i=l i=n+l 

Proof. This is immediate from Corollary 24, Lemmas 22, 23 and Proposition 21. □ 
6. Some properties of the invariant eigendistributions on the symplectic 

SPACE. 

The group S acts on Sj via the adjoint action. Hence, we have the permutation repre- 
sentation on the Schwartz space S{Si): 

7r(s)0(a;) = (l){s-\x) (s G S, a; e Sj, </> G S{sj)). (76) 

with the derivative 

'!r{z)(j){x) = ^(f){exp{-tz).x)\t=o = d{-[z,x])(p{x) {z e5o,x e St, G S{$j)). (77) 

As mentioned in the Introduction, a distribution / e S*{Si) shall be called an invariant 
eigendistribution if 

7r(s)/ = / (seS) (78) 

and 

T{z)^f^^{z)f (zeUis^f), (79) 

where 7 : W(So)^ — > C is an algebra homomorphism. Then (79) is the system of equations 
explained in the introduction. Notice that if S is connected then (78) is equivalent to 

d{[z, x])f{x) = fix) {z e 5o, X e sj). (80) 

Thus in this case / is an invariant eigendistribution if it satisfies the two systems of 
differential equations (79) and (80). 
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Let US identify Sj with the dual Sj by 

x'{x) = {x,x') {x,x' G S-). 
This leads to an identification of the cotangent bundle to Sj with Sj x Sj: 

Lemma 26. The characteristic variety of the system of equations (79) and (80) is equal 
to the set of all the pairs (x, y) & SjX Sj such that the anticommutant {x, y} = and y is 

nilpotent. 

Proof. The principal symbol of the differential operator (80) may be computed as follows 

lim t-ie-^*<^'^>9([^,x])e^*<^'^> = i{[z,xly). 

Hence, the characteristic variety of the joined system (80) consists of pairs {x,y) such 
that 

([z,x],y)=0 (^eso). 

In other words, y is orthogonal to the space [%, a;]. By [Prz06, Lemma 3.5] this last 
condition is equivalent to the vanishing of the anticommutant, {x, = 0. But formula 
(7.22) in [Prz91] implies that the characteristic variety of the system (79) consists of the 
pairs (x, y) such that y is annihilated by all the non-constant, complex valued, S-invariant 
polynomials on Sj, i.e. y is nilpotent. □ 

An element x e Sy called regular if the semisimple part of its Jordan decomposition 
is regular. We shall denote by Sj^^ C sj the subset of all the regular elements. 

Lemma 27. The characteristic variety of the system of equations (19) and (80) over the 
set of the regular elements Sj^^ C Sj is equal to Sj^^ x {0}. 

Proof. We may assume that (S,5) is a complex supergroup. Let x e Sj^^ and let 
x — Xs -\- Xn he Jordan decomposition of x. Then, by [Prz06, Theorems 6.10(b) and 
4.4(a)], the supergroup (S^'',Sy) is isomorphic to the direct product of several copies of 
the supergroup whose underlying dual pair coincides with (GLi(C), GLi(C)), and one 
copy of (Oi(C),Sp2jC)),ifx,7^0. 

Let y e Sy be nilpotent and anticommute with x. We need to show that y — 
It will suffice to show that y is semisimple. Notice that y commutes with = x1 -\- 

we may assume that the underlying dual pair is either 
(GLi(C), GLi(C)) or (Oi(C), Sp2„(C)). In the first case the defining module V = Vq© Vy, 
with dim Vq = dim Vy = 1. Let Vq G Vq and vi G Vy. We may assume that 

x{vo) = ^Vi and x{vi) = ±C,Vo, 

where ^ G C^. Then the anticommutant of x consists of elements x' such that 

x'{vo) = C'vi and x'{vi) = T^'vo, 

where ^' G C, see [Prz06, (13.47)]. In particular x', and hence y, is semisimple. 
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In the second case x is any non-zero element of S-. Recall the following formula 

Since the set of all the commutators [z, x] coincides with Sj (because the set of the non- 
zero elements of Sj is a single Sp2„(C)-orbit) and {x,y} = 0, we see that y is orthogonal 
to Sj and hence equal to zero. □ 

By combining Lemmas 26 and 27 with [Hor83] we obtain the following corollary. 

Corollary 28. The restriction of an invariant eigendistrihution on Sj to Sj^^ is a smooth 
function. 



34 



M. MCKEE, A. PASQUALE, AND T. PRZEBINDA 



Appendix A 



In this appendix we verify the assumptions of [Ros90, Corollary 5.4, page 283] for a 
minimal non-zero nilpotent orbit in a real symplectic Lie algebra 

A B 
C -A^ 



5p2n 



X 



(A.l) 



Let f) be the elliptic Cartan subalgebra consisting of the matrices X — X{xi^X2, ■ ■ ■ , 

as in (A.l), with A = B diagonal with diagonal entries xi, X2,. . . , x„, and C — —B. 

Also, let t C 5p2„(]R) be the subset of the skew-symmetric matrices. 

The set of regular elements \f'^^ in \) consists of the X such that the Xj are all distinct 
and non-zero. Let l)+ C be the Weyl chamber defined by the condition Xi > X2 > 
• • • > x„ > 0. For t > let 



a{t) 



Then a{t) e Sp. 



2n\ 



and 



a{t)Xa{t)-^ 





/ t 
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Hence, 



lim r^a{t)Xa{t)-^ = 

t— >-oo 
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(A.2) 
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If Xi > then (A. 2) belongs to one minimal nilpotent orbit O, and if xi < it belongs to 
the other nilpotent orbit equal to — O. Let Oc be the complexification of O. This is the 
unique minimal nilpotent orbit for the complex symplectic group Sp2„(C). We see that 

O = Oc n (closure of Sp2„(M).R+X) (X e f)+). (A.3) 

This verifies [Ros90, Hypothesis (O), page 280]. 

Let us choose a positive root system for the roots of f) in so that 7rt/(,(X) = 
Y[i<j<k<n('^^k — i^j)- The Weyl group W{sp2n(^)-,'^) consists of all the permutations 
and the sign changes of the Xj. This group acts on C[[)], the space of the complex valued 
polynomials on f), in the obvious way. The representation poc of W^(-Sp2ri(Il^), I)) attached 
to the orbit Oc via Springer correspondence may be realized on the subspace of the poly- 
nomials generated by Tr^/f,. (One way to see it is use Wallach's theorem [Wal93] saying that 
POc generated by the product of fxo, the Fourier transform of the invariant measure on 
O restricted to f)"*", times T^Bxi2n{'S-)h product of all the roots of P) in sp2n(Il^)- The Fourier 
transform is well known and is equal to a constant multiple of the function chc given in 
(4), see for example [PrzOO, proof of Proposition 9.3]. In particular the restriction to f)+ 
coincides with the reciprocal of the product of the long roots. Therefore /io|f)+7rsp2„(R)/i,|i,+ 
is a non-zero constant multiple of TTf/;,.) The one-dimensional sign representation of the 
subgroup W{t, [)), which coincides with the group of all the permutations of the Xj, is 
contained in po^ with multiplicity one. In fact the sign representation is realized on the 
complex hue through tt^/;,. This verifies the assumptions of [Ros90, Corollary 5.4]. 



Appendix B 

Here we verify Proposition 17. Recall that our dual pair acts on a defining module 
V = Vq©Vy, which is a finite dimensional Z/2Z-graded vector space over D = M, C or H. 
For z G Sj, the pair {z.M) is called decomposable if there are two Z/2Z-graded non-zero 
subspaces V, V" C V preserved by z (and orthogonal if the corresponding dual pair is of 
type I) such that \J — ® V". Otherwise {z, V) is called indecomposable. 

We shall verify Proposition 17 under the assumption that the nilpotent element z & Sj 
is such that {z, V) is indecomposable. (We shall see at the end of this appendix that this 
implies the result for general nilpotent elements.) 

For dual pairs of type I such elements are classified in [DKP05, Theorem 5.2]. There 
are seven cases to consider (Case La - Case I.g below). For pairs of type II there are two 
cases, see [Prz06, Theorem 5.5] (Case II. a and Case II. b below). (Some misprints which 
occur in [DKP05, Theorem 5.2] have been corrected in [Prz06, Theorem 5.4]. There is 
one remaining misprint in [Prz06, Theorem 5.4] which we correct in what follows.) In 
each case we describe the graded vector space V, the automorphism 9 (39), the Cartan 
subspace \)j C Sj, the abehan Lie algebra a C and the decomposition of z into root 
spaces for the action of a. 

We begin with the dual pairs of type I. In this case there is a possibly trivial involution 
D9a— >-aeDon the division algebra D. We shall assume that the space Vq is equipped 
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with a non-degenerate hermitian (or symmetric) form ( , )o and that Vj with a non- 
degenerate skew-hermitian (or skew-symmetric) form ( , )y. Let ( , ) = ( , )o ® ( ; )t- 
Then, in particular, x G End(V) belongs to Sj if and only if [xu, v) = {u, sxv) for all 
G V, where s{veven + Vodd) = Veven — Vodd- Recall that the signature of a hermitian 
(or symmetric) form is the difference of the dimension of a maximal subspace where the 
form is positive definite and the dimension of a maximal subspace where the form is 
negative definite. Wc shall write sgn((-, = 1 if the form (■, •)q has positive signature, 
sgn((-, ■)()) = —1 if it has negative signature and sgn((-, = if the signature is zero. 
In each case we'll describe an element T G S such that for a specific basis of V consisting 
of vectors Vk, {Tvk,Vk) > 0. Consequently the conjugation by T coincides with the 
automorphism 9 (39), unless the involution D9a— >-aGDis trivial. In that case 9 
is the conjugation by T composed with the conjugation on s induced by the complex 
conjugation on V which leaves the vectors v^. fixed. Set 



Then, in particular, S{k + 1) = (— 1)^5(A;), if m is divisible by 4 then S{m + 2 — k) — 
— {—l)^d{k), if m is even but is not divisible by 4 then d{m — k) = {—l)^~^^d{k) . 




(B.l) 
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Case La: S = Op+i,p x Sp2p(M), 02p+i(C) x Sp2p(C), Up+i,p x Up,p or Sp^+i^p x O^^ 



m e 4Z, m > 0; 

m 
fe=0 

Vfc = z'^vq 7^ 0, < A; < m, 2;fm = 0; 

Tfi 

{vk, vi) =0 if I j^m-k, {vk, v„,-k) = S{k)6{—) sgn((-, 

where, by definition, sgn((-, ■)q) = 1 

if D = C and the involution D^a^aeDis trivial; 

Here, 

Tvk = tkVrn-k, h = {-!)'' S{k)S{—) sgn((-, 0<k<m. 
The Cartan subspace l)x consists of the linear maps x defined by 

XV2j = XjV2j+l, XV2j+l = XjV2j, XV m. = 0, 

if D 7^ H, then xj E D, 

TTl 

if D = H, then Xj e C ^ the centrahzer of z in H, <2j < —. 

This is the direct sum of the indecomposable Cartan subspaces which occur in [Prz06, 
Theorem 6.2(a)]. 

The Lie algebra a consists of the hnear maps a defined by 

avk — Ofeffe, Ofe e R, < A; < m, 

m 

= -cim-k, < fc < — , = 0, 

TTl 

a2j = a2j+i for < 2j < — , 
For < k < Y define a hnear map z{k) by 

z{k)Vk = Vk+l, z{k)Vm-k-l = Vm-k-, 

z{k)vj — 0, j ^ {k, m — k — 1}. 

Then z = X]fc=o ^(^) ^ -St a e o, [a, = (a^+i — ak)z{k). In particular 

2;(ei'en) G YlT=i ^ii ^'^'^ z{odd) G rii. 

For a fixed j with < 2j < ^ and for i > define h^h{t) G S by 

bVm-2j—t ^Vm-2j, bVm-2j-l — tVm-2j-l, 

bvk = for k ^ {2j, 2j + 1, m - 2j, m-2j - 1}. 

Also, let X = x(t) G f)Y be such that xj = and Xi — Q for i ^ j. Then lim^^o ^'a;^"^ = 
z{2j). Hence, zi — Y^- z{2j) is the limit of elements of the L-orbits passing through 
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Case Lb: S = Up,p x Up+i,p or Sp^^ x 0^(2^+1) 

m e 4Z, m > 0, D 7^ R, the involution D 9 a ^ a e D is not trivial; 

m+l 
fc=l 

= 7^ 0, < < m, zvrn+i = 0; 
(wfcjt;/) = if / 7^ m + 2 - A;, {vk,Vm+2-k) = S{k){vi,Vm+i), 

Tfl 

(vi, Vm+i) = i sgn(-i(-, ■)j)S{l + y), if D = C, 
(^i,^m+i) if e = H; 

Here, 

Tvk = tkVm+2-k, tk = {-l)''S{k){vi, Vm+l), 1 < k <m+l. 
The Cartan subspace f)x consists of the linear maps x defined by 

XV2j+l — XjV2j+2, XV2j+2 — XjV2j+l, XVrn±2 — 0, 
XVm+l-2j — ~XjVm-2j, XVjn-2j — XjVjn+l-2j, 

TTX 

Xj & C — centralizer of i in D, 1 < 2j + 1 < — . 

This is the direct sum of the indecomposable Cartan subspaces which are isomorphic to 
those which occur in [Prz06, Theorem 6.2(a)]. (One has to adjust the sesquihnear forms 
To and Ti hsted in [Prz06, Theorem 6.2(a)] in order to get the forms ( , )o and ( , )i we 
are working with here.) 

The Lie algebra a consists of the linear maps a defined by 

avk = ttkVk, afc G M, 1 < k < m + 1, 

m + 2 

Qk — —am+2-ki 1 < k < — - — , am+2 — 0, 

Z 2 

a2j+i = a2j+2 for 1 < 2j + 1 < — . 
For 1 < A; < y define a linear map z{k) by 

z{k)Vk = Vk+l, Z{k)v,n+l-k = Vm+2-k, 

z{k)vj = 0, j ^ {k, m + l - A;}. 

m 

Then z = "^^=1 z{k), z{k) G Sj and for a G a, [a,z{k)] — (a^+i — ak)z{k). In particular 
z{odd) G X^i^i ki z{even) G rii. 

For a fixed j with 1 < 2j + 1 < f and for i > define b = b{t) G S by 

bV2j+l = tV2j+l, bV2j+2 = t~^V2j+2, 
bVm+l-2j = t ^Vm+l-2j, b^'m-2j = tVm-2j, 

bvk = Vk for k ^ {2j + 1, 2j + 2, m + 1 - 2j, m - 2j}. 

Also, let X = x{t) G [)- be such that xj = and ajj = for i 7^ j. Then lim^o^a;6~^ = 
z{2j + 1). Hence, z\ = z{2j + 1) is the limit of elements of the L-orbits passing through 
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Case I.c: S = Qp,p x Sp2(p+i)(R), 02p(C) x Sp2(p+i)(C), Vp+i,p x Up,p or Sp^^^^p x O^^ 



m e 4Z, m > 0, D 7^ H, the involution D9a^aeDis trivial; 

m+1 

V = J](D^;fe + D^;^); Veven, ^^Len ^ Vq; e V^; 

k=l 

Vj+i = z^vi ^ 0, v'j^-y = z^v[ 7^ 0, <j <m, zvm+i = zv'^+i = 0; 

{Vk, ^^m+2-fc) = {Vk, Vm+2-k) = -S{k), 1 < A) < m + 1 

and all other pairings are zero; 

Here, 

4 = {-l)''6{k), ti = -{-l)''6{k), l<k<m + l. 
The Cartan subspace i)i consists of the linear maps x defined by 

XV2j+l = XjV2j+2, XV2j+2 = XjV2j+l, XVm+l = 0, 
^^m+l-2j ~ ^^i'^m-2j5 ^'^m-2j ~ ^j^m+l-2j) ^'^1 ~ 

Xj eBi, 1 < 2j + 1 < m + 1. 

This is the direct sum of the indecomposable Cartan subspaccs which arc isomorphic to 
those which occur in [Prz06, Theorem 6.2(a)]. (One has to adjust the sesquilinear forms 
To and Ti listed in [Prz06, Theorem 6.2(a)] in order to get the forms ( , )q and ( , )i we 
are working with here.) 

The Lie algebra a consists of the linear maps a defined by 

avk = atVk, a^4+2-fc = -akv'm+2-k^ Ofe e M, 1 < k < m, 
avm+i = 0, av[ = 0, 
a2j+i = a2j+2 for 1 < 2j + 1 < m + 1. 
For 1 < /c < m define a linear map z{k) by 

z{k)vk = Vk+i, z{k)v'^^i_,, = t;^+2-fe, 

z{k)vj = 0, z'{k)v'^^i_j = 0, k. 

Then z = J2T=i -^(^) ^ -^T a & a, [a,z{k)] — {ak+i — ak)z{k). In particular 

z{odd) G ki ^iid z{even) G rii. 

For a fixed j with 1 < 2j + 1 < m + 1 and for t > define b = b{t) G S by 

bV2j+l = tV2j+l, bV2j+2 = t~^V2j+2, 
bVm+l-2j = t ^Vm+12ji bVm-2j = tVm-2j, 

bvk = Vk for k ^ {2j + 1, 2j + 2, m + 1 - 2j, m - 2j}. 

Also, let X = x{t) G i)i be such that Xj = and = for i 7^ j. Then limj^o bxb~^ = 
z{2j + 1). Hence, zi = J^j ^C^J + 1) is the limit of elements of the L-orbits passing through 
f)T- 
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Case I.d: S = Up,p x Up,p_i or Sp^^ x 02(2p_i) 



m e 2Z \ 4Z, m > 0, D 7^ R, the involution D 9 a ^- a e D is not trivial; 

rn 



fc=0 



Vk = z vo 0,0 < k < m, zvm = 0; 
{vk, vi) = a I ^ m - k, {vk, Vm-k) = S{k)ic, c = ±1. 



Here, 



Tvk = tkVm-k, tk = {-l)'^S{k)ic, <k <m. 
The Cartan subspace f)Y consists of the linear maps x defined by 

XV2j = XjV2j+l, XV2j+l = XjV2j, XV in = 0, 

Tfl — 2 

Xj — the centralizer of i in D, < 2j < — ^ — , 

1 . 

1 , 

This is the direct sum of the indecomposable Cartan subspaces which occur in [Prz06, 
Theorem 6.2(a) and (b)]. 

The Lie algebra o consists of the linear maps a defined by 

avk — ttkVk, Ofe e R, < A; < m, 

m 

Ofe = -cim-k, 0<k<—, am = 0, 

TTl 

a2j = a2j+i for 0<2j < —, 
For < A; < define a hnear map z{k) by 

z{k)Vk = Vk+l, z{k)Vm-k-l = Vm-k, 

z{k)vj = 0, J ^ {A;, m — k — 1}. 

Then z — '^k=o ^i^) ^ -^T a e a, [a, = (ofe+i — ak)z{k). In particular 

2;(ei'en) e Yl^i z{odd) e rii. 
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Set p = f , M = ^(1 + i) and for t e define 

bvp-i = tvp-i, bvp = uvp, bvp+i = t~^Vp+i, 
bvk =Vkiik^{p-l,p,p + 1}. 

Then 6 G S and 

bxb~^Vp_i = t~^u^=Vp, bxb~^Vp — ^={tVp^i + t~^ivp+i), 
V 2 V 2 

bxb~^Vp+i = —itu^^Vp. 

v2 

Choose Xp = —iu\f2t. Then z{j) — 1) = \\m.t^Qbxb~^ . Thus z{p — 1) £ ^oT the hmit of 
the semisimplc elements of [qt- 

For a fixed j with < 2 j < and for t > define b = b{t) G S by 

bV2j = tV2j, bV2j+l = t~^V2j+l, 

bVm-2j = t ^Vm-2j, bVm-2j-l = iVm-2j-l, 

bvk = Vk for k ^ {2j, 2j + 1, m - 2j, m - 2j - 1}. 

Also, let X = x{t) G [)x be such that Xj = t"^ and = for i 7^ j. Then limt^obxb~^ ~ 
z{2j). Hence, Zi = z{2j) is the limit of elements of the L-orbits passing through 
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Case I.e: S = Op,p„i x Sp2p(M), 02p-i(C) x Sp2p(C), Up,p-i x Up,p orSppp^^ x 

(This is the only case where (1) or (2) occurs. It happens when the involution B) 3 a ^ 
a e D is trivial.) 

m e 2Z \ 4Z, m > 0; 

m+l 

V = ^ BVk, Veven £ Vq, Vodd £ Vy; 
k=l 

Vk = z^vi 0,0 < k < m, zVm+i = 0; 

{vk,vi) = 0iil^m + 2-k, {vk,Vm+2-k) = S{k){vi,Vm+i), 

{vi, Vm+l) = (^(^^^^^) sgn((-, 

Here, 

TVk = tkVm+2-k, tk = {-!)'' S{k){vi, Vm+l), < k < Ul. 

The Cartan subspacc i)j consists of the linear maps x defined by 

XV2j-l = XjV2j, XV2j = XjV2j-l, 

XVm+3-2j = —XjVm+2-2j, XVm+2-2j = XjVm+3-2j, 

m -\- 2 

X j e C = the centralizer of i in D, < 2j < — - — , 

if the involution D9a— >-aeDis trivial, then xvrn±2 — 0, 

if the involution D9a— )-aeDis not trivial, then 

XVm+2 — Xm.+2—=[VVl +lVmL+2)-, X—;=[VV1 + IV 121 +21 = Xm + 2Vm+2, 

— — ^2 ' ' V2 ' 2+ ' — ^' 

X m+2 e — ~ 1)^) 
^ V 2 

This is the direct sum of the indecomposable Cartan subspaces which occur in [Prz06, 
Theorem 6.2(a) and (b)]. 

The Lie algebra o consists of the linear maps a defined by 

avk — Ofeffe, Cfe e R, 1 < < m + 1, 

m + 2 

Ofe = —0>m-\-2-k-i 1 < A; < , 022+2 = 0, 

oaj-i = for < 2j < — - — , 
For 1 < A; < define a hnear map z{k) by 

z{k)Vk = Vk+l, z{k)Vm+l-k = 

z{k)vj = 0, j ^ {A;, m + 1 — A;}. 
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m + 2 

Then z = J2k=i ^(^) ^ -^T for a E a, [a,z{k)] = (a^+i — ak)z{k). In particular 

z{odd) e YJU ki. e toT and z{2j) e til for < 2j < 

Assume that the involution D9a— >-aeDis not trivial. Set p = u — ^(1 + 
and for i e define 

bvp_i = tvp_i, bvp = uvp, bvp+i = t~^Vp+i, 
bvk ^Vkiik^{p- + 1}. 

Then 6 e S and 

bxb~^Vp_i = t~^u^=Vp, bxb~^Vp = u~^^={tVp_i + t~^ivp^i), 
V 2 V 2 

bxb'^Vp+i — —itu—j^Vp. 

V2 

Choose Xp = —iu\pit. Then z{j) — 1) = limt^o bxb~^. Thus z{p — 1) G Iqt is the limit of 
the elements of the L-orbit passing through 

If the involution ©9a— )-aeDis trivial, then the only semisimple element in Iqj is 
0. The restriction of S to the span of Vp and fp+i is isomorphic to Oi x Sp2(IR), if 
D = M and to Oi x Sp2(C), if D = C. The complement of in [qt is a single nilpotent 
orbit under the action of this group. 

For a fixed j with < 2j < ^ and for i > define b = b{t) G S by 

bV2j-l = tV2j-l, bV2j = t''^V2j, 

bVm+3-2j — t ^Vm+3+2j, bVm+2-2j — tVm+2-2j , 

bvk = Vk for k ^ {2j - 1, 2j, m + 3-2j,m + 2- 2j}. 

Also, let X = x{t) e f)i be such that Xj — and Xi — ior i ^ j. Then limt_^o bxb~^ — 
z{2j — 1). Hence, zi — z{2j — 1) is the hmit of elements of the L-orbits passing through 

m+2 

Notice that the same holds for the nilpotent element z — ^^=2 ^(^)) ^{^) ^ -^T- 
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Case I.f: S = 0^,^ x Sp.^^^.^jR) or 0,p{C) x Sp2(p+i)(C) 

m e 2Z \ 4Z, m > 0, B^M, the involution D9a^^aeDis trivial; 

m 

V = 5^(D^;fe + D^;;;); Veven, ^^Len ^ Vq; v^rfrf e Vx; 

fc=0 

Vj = z^vo ^ 0, Vj = z^v'q 7^ 0, 0<j<m, zVm = zv'^ = 0; 
K,i;^_fe) = (5(A;), (wfc,w™_fc) = -6{k), < A; < m 
and all other pairings are zero; 

Here, 

tk = (-1)^5(A;), 4 = -{-l)''6{k), 0<k<m. 
The Cartan subspace i)j consists of the linear maps x defined by 

XV2j = XjV2j+l, XV2j+l = XjV2j, XVjn = 0, 

xjeB, o<j<-. 

This is the direct sum of the indecomposable Cartan subspaces which are isomorphic to 
those which occur in [Prz06, Theorem 6.2(a)]. (One has to adjust the sesquilinear forms 
To and Ti listed in [Prz06, Theorem 6.2(a)] in order to get the forms ( , )q and ( , )i we 
are working with here.) 

The Lie algebra a consists of the hnear maps a defined by 

avk = akVk, av'^_k = -akv'^_^, Ofe e R, < A; < m, 

avm = 0, avQ = 0, 

a2j — Cb2j+i for < 2_7 < m. 

For < A; < m define a linear map z{k) by 

z{k)vk = Vk+u z{k)v'^_k_i = v'^_k, 

z{k)vj = 0, z(k)v'^_^_, = 0, i ^ k. 

Then z = Yl^=o ^ik)^ z{k) e Sj and for a e a, [a,z{k)] = (a^+i — ak)z{k). In particular 
z{even) e YllLi z{odd) e rii. 

For a fixed j with < 2 j < f and for i > define h = h{t) G S by 

hV2j = tV2j, bV2j+l = t~^V2j+l, 
bVm-2j—t ^Vm-2j-i bVm-2j-l — tVm-2j-l-i 

bvk = Vk for k ^ {2j, 2j + 1, m - 2j, m-2j - 1}. 

Also, let X — x{t) G f)i be such that xj = and = for i ^ j. Then limf^o hxh~^ — 
z{2j). Hence, zi — Y^- z{2j) is the limit of elements of the L-orbits passing through ^j. 
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Case I.g. S = Op,p x Sp2p(M), Up,p x Up,p or Sp^^^ x O^^ 
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me 21 + 1; 

m 
A;=0 

Vj = z-^vo 7^ 0, v'j^-^ = z^v[ 7^ 0, <j <m, zVm = zv'^^-^ = 0; 
K, t^m+i-fe) = ^{k), (v^+i, v^.fc) = + l)5(m), < A; < m 
and all other pairings are zero; 

Here, 

4 = 5(A'; + 1), = 5(m + 1 - A;), < A; < m. 
The Cartan subspace ()y consists of the linear maps x defined by 

XV2j = XjV2j+l, XV2j+l = XjV2j, 

if the involution D3a— )-aG©is trivial, then Xj G ©, 

777- 

otherwise G C C D, < j < — . 

This is the direct sum of the indecomposable Cartan subspaces which are isomorphic to 
those which occur in [Prz06, Theorem 6.2(a)]. (One has to adjust the sesquilinear forms 
To and Ti listed in [Prz06, Theorem 6.2(a)] in order to get the forms ( , )q and ( , )i we 
are working with here.) 

The Lie algebra a consists of the hnear maps a defined by 

avk = a^Vk-, av'^+x^k = -0'kv'^+x-k-> «fc ^ < /c < m, 
O'lj — a2j+i for < 2_7 < m. 
For < A; < m define a linear map z{}i) by 

z{k)v^ = 0, z{k)v'^^^_^ = 0, ^■ 

Then z — ^}^Zq z{)z)^ z{k) G Sj and for a G a, [a, z(A;)] = (ofe+i — ak)z{k). In particular 
z{even) G ki ^^i^ z{odd) G rii. 

For a fixed j with < j < ^ and for i > define h — h{t) G S by 

hV2j = tV2j, hV2j+\ = t~^V2j+\, 

bVm+l-2j = i ^'ym+l-2j, bVm-2j — tVm-2j , 

bvk = Vk for k i {2j, 2j + l,m + l- 2j, m - 2j}. 

Also, let X = x{t) G i}i be such that Xj = and = for i ^ j. Then limj^o hxh~^ — 
z{2j). Hence, zi = z{2j) is the limit of elements of the L-orbits passing through ()y. 
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Case I."V ": S = Q2 x Sp^lR); 



D = R; 

V = Di;i + Bv2 + Bv3 + BV2; 

zvi = V2, ZV2 = V3, ZV3 = 0, ZV2 = 0; 

{vijV^) — 1, {v2,V2) — iv2,V2) — —1 and all the other pairings are zero; 
The Cartan subspace f)i consists of the linear maps x defined by 

X^{V2 + V2) = Xi{vi + V3), 

v2 

X-j={-V2 + V2) = Xi{vi - V3), 
XVi = Xi\/2V2, 

XV3 — —X\\p2v2- 

where x\ e M, or equivalently, 

XV2 = X1V3, Xv'2 = XiVi, 
XVi = X1V2, XV3 = —xiv'2. 

This is the Cartan subspace which is isomorphic to the one which occurs in [Prz06, 
Theorem 6.2(c)]. 

For i > define h = h{t) G S by 

hvi — tvi, bv2 — V2, bvs — t'^Vs, bv2 — v'2- 

Also, let xi — t. Then limj-^o bxb"^ — z. 
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Case I."V V": S = O2 X Sp4 D (Qi x Sp^) x (Oi x Sp^) 
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D = R or C with trivial involution; 

V = (Bvi + Bv2 + Bvs) + (Bv'i + Bv'^ + Bv's) ; 

zvi — V2, ZV2 — V3, ZV3 — 0, zv[ — V2, ZV2 — v'^, zv'^ — 0; 

{vi, V3) — {v[, v'^) — 1, {v2, V2) — {v2, V2) — —1 and all the other pairings are zero; 

In this case there is only one, up to conjugation, Cartan subspace i)i C Sj. If D = R, then 
f)Y may be realized as the space consisting of linear maps x defined by 

a;--^(w2 + V2) = xi{vi + V3), 

X^{-V2 + V2) = Xi{vi - V3), 

XVi = Xi 

xv^ = —X1V2V2. 
where xi e R, see [Prz06, Theorem 6.2(c)]. Equivalcntly, 

XV2 = X1V3, Xv'2 = XiVi, 

XVi = X1V2, XV3 = —xiv'2. 

If D = C, then is isomorphic to the Cartan subspace comprised of elements which 
occurs in [Prz06, Theorem 6.2(a)]. In any case the kernel of a non-zero element of 1)^ is 
contained in Vy. However any semisimple orbit in Sj passes through a Cartan subspace. 
Hence, the kernel of any non-zero semisimple element of Sj is contained in Vj. Let x ^ Sj 
and let x = Xs + .t„ be Jordan decomposition of x, see [Prz06, ]. Since Xn G Sj commutes 
with Xs G Sj we see from the above that Xn = 0. Thus x is either nilpotent or semisimple. 
Since there are finitely many nilpotent orbits in By, our given nilpotent z is the limit of 
elements of the S-orbits passing through ^j. Similar argument applies to the previous 
case too. 
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Next we consider the dual pairs of type II. As for the dual pairs of type I, x E End(V) 
belongs to Sj if and only if {xu,v) — {u,sxv) for all u,v & V, where s{veyen + Vodd) — 
Veven ~ ^odd- Also, if B 7^ K, then there is a nontrivial involution on D. In any case we'll 
describe a basis on V consisting of vectors Vk- Then there is a positive definite hermitian 
(or symmetric if D = M) form r]{ , ) on V defined by the condition r]{vk, Vk) = 1- This form 
determines an involution End(V) 9 a; — )■ e End(V) defined by r]{xu,v) = r]{u,x^v). 
Then 9{x) — —x^ if x e % and 9{x) — sx^ if x e Sj. 

Case Il.a: 

m e Z, m > 0; 

m 

V = ^ BVk, Veven £ Vq, Vodd £ Vy; 
fc=0 

Vk — z^vq 7^ 0, < /c < m, zvm — 0; 
If m is odd, then the Cartan subspace \)j consists of the linear maps x defined by 

XV2j = XjV2j+l, XV2j+l = XjV2j, 

if D ^ H, then xj e D, 

if D = H, then xj eCCM, < 2j < m. 

If m is even, then the Cartan subspace f)Y consists of the linear maps x defined by 

XV2j+l = XjV2j+2, XV2j+2 = XjV2j+l, XVq = 0, 

if D ^ H, then xj G D, 

if D = H, then xj eCCM, < 2j < m. 

These are the direct sums of the indecomposable Cartan subspaces which occur in [Prz06, 
Theorem 6.2(e)]. 

If m is odd, then the Lie algebra a consists of the linear maps a defined by 

avk — ttkVk, Ofc e R, < k < m, 
0'2j — 0'2j+i for < 2_7 < m. 

If m is even, then the Lie algebra a consists of the linear maps a defined by 

dVk = cikVk, ttk eM., 1 < k < m, avo = 
a2j+i = a2j+2 for < 2j < m. 

For < k < m define a linear map z{k) by 

z{k)vk = Vk+i, 
z{k)vj = 0, k. 
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Then z = Yl^=o ^i^^)^ ^{^) ^ ^^"^ ^^^^ a G a, [a,z{k)] = (a^+i — ak)z{k). If m is 
odd, then z{even) e Yl^i ki z{odd) e rii. If m is odd, then z{odd) e Yl^i ki 
z{even) e rii. 

Suppose m is odd. For a fixed j with < 2j < m and for i > define 6 — b{t) G S by 

bv2j = tv2j, bv2j+i = i~^^;2j+i, 
= Vk for A; ^ {2j, 2j + l}. 

Also, let X = x{t) e f)Y be such that Xj — and = for z 7^ j. Then limt_>.o hxh~^ — 
z{2j). Hence, zi — z{2j) is the limit of elements of the L-orbits passing through f)j. 
Suppose m is even. For a fixed j with < 2j < m and for i > define 6 = G S by 

bV2j+l = it'2j+l, &'y2j+2 = t~^V2j+2: 

bvk = for k i {2j + 1, 2j + 2}. 

Also, let X — x{t) e l)x be such that = t'^ and = for i 7^ j. Then lim^-^o bxb"^ — 
z{2j + 1). Hence, zi — 2;(2j + 1) is the limit of elements of the L-orbits passing through 

Case II. b: 



m e Z, m > 0; 

m+l 

fc=l 

= -z^t^i ^ 0,0 < k <m, zvm+i = 0; 
If m is odd, then the Cartan subspace f)x consists of the linear maps x defined by 

XV2j+l = XjV2j+2, XV2j+2 = XjV2j+l, XVm+1 = 0, 

if D 7^ H, then xj G D, 

if D = H, then Xj eCCM, < 2j < m. 

If m is even, then the Cartan subspace f)^ consists of the linear maps x defined by 

XV2j+l = XjV2j+2, XV2j+2 = XjV2j+l, 

if D 7^ H, then xj E D, 

if D = H, then Xj eCCM, < 2j < m. 

These are the direct sums of the indecomposable Cartan subspaces which occur in [Prz06, 
Theorem 6.2(e)]. 

If m is odd, then the Lie algebra a consists of the linear maps a defined by 

avk — Ofeffc, Ofe e M, < A; < m, aVm+i — 0, 
a2j+i = a2j+2 for < 2j < m. 
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If m is even, then the Lie algebra a consists of the Unear maps a defined by 

avk = akVk, Ofc G M, 1 < k < m, 
a2j+i = a2j+2 for < 2j < m. 

For < k < m define a hnear map z{k) by 

z{k)vk = Vk+i, 
z{k)vj = 0, j ^ k. 

Then z = '^^Zo ^{^) ^ '^T for a E a, [a,z{k)] = (a^+i — ak)z{k). Then 

z{odd) e kj and z{even) e rii. 

For a fixed j with < 2j < m and for i > define b — b{t) G S by 

bV2j+l = tV2j+l, bV2j+2 = t~^V2j+2, 

bvk = Vk for k i {2j + 1, 2j + 2}. 

Also, let X = x{t) G i)j be such that Xj = t"^ and a;, = for i 7^ j. Then lim^^o bxb~^ = 
z{2j + 1). Hence, zi = z{2j + 1) is the limit of elements of the L-orbits passing through 

By combining the cases Case La - Case II. b we see that Proposition 17 holds if {z, V) is 
indecomposable. Every nilpotent {z, V) is a finite direct sum of indecomposable nilpotents 
{zi, Vj), [DKP05, Definition 3.14]. Suppose S is not isomorphic to an ortho-syniplectic pair 
(Op.q X Sp2„(IR) or Op(C) X Sp2„(C)). Then each {zi, Vj) has the same property (the group 
S|vi is not ortho-symplectic) and the Cartan subspacc i)j can be defined as the direct 
sum of the Cartan subspaces constructed for each {zi, Vj). Also the involution 9 may be 
extended from each s(Vi) to s(V). Hence, Proposition 17 holds for {z,\/), which is the 
sum of the (zi, Vj). 

Suppose S is an ortho-symplectic pair. Let 

{z, V) = (^1, Vi) ® • • • © {Zr, V,) ® (0, Vo) 

be the decomposition into irreducibles, where each Zi is non-zero. As we checked in Case 
I.c - Case I.g, the proposition holds for each (^j, Vj). However this docs not automatically 
imply the proposition for the sum, because the sum of the individual Cartan subspaces 
constructed for the {zi, Vj) may be too small (could be zero). This problem does not occur 
if dimji\/{Q is even for each i. Thus we may assume that dimB V-q is odd for each i. 

If dimo\/{Q > dimj],\/,jj, then we are in Case La, and if dimoV^Q = dimeN/jj, then we 
are in Case I.g, the proposition holds for the sum of such {zi, Vj) with the Cartan subspace 
equal to the sum of the individual Cartan subspaces. 

Thus we may assume that dining iQ < dimn\/{i for each i. Thus each {zi,\/i) is as in 
Case I.e. Here we combine Case I.e with either Case I."V V" or Case I."V ." to construct 
the Cartan subspace and the involution 9 for the sum. 
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Appendix C 

In this appendix we conclude the proof of Theorem 19. Let (S,5) be a supergroup 
associated with a dual pair (U„,Un) or (GL„(D), GL„(D)) with D = R, C or H. Let 
X — Xs + Xn he the Jordan decomposition of an element x e Sj. We suppose that Xs ^ 
and Xn 7^ 0. Moreover, we may assume that Xg belongs to a isotypic Cartan subalgebra 
built up from indecomposable blocks as in [Prz06, Theorems 5.2(b) and 5.3]. 

S —— X U^. 

V = Vo X Vt with V, = ELi ^^a,k {a e Z/2Z) . 

Xg — Xs{a) — with a e C \ {0} and I the n x n identity matrix. 

Xn — (^^^ q'^ with w e M„(C) and w* = iw*. 

Since XnXg — XgXn, the matrix w must satisfy w* — w. Hence x^ = E Sq — 

Un X u„. Since a;^ is nilpotent, so must be w^. But every matrix in u„ is diagonalizable. 
Hence w'^ — 0. Taking traces, we obtain itic{iD*w) — tr(w^) = 0, which implies w — 0. 
This shows that in this case every element in Sj is either nilpotent or semisimple. 



S = GL„(C) X GL„(C): 

w' 



V and Xs = Xs{a) are as in the previous case. Xn = ( ^ ) with w,w' G M„(C). 



Prom XnXs — XsXn we obtain that w' — w. Thus x — Xs + Xn 



al + w 
al + w 

By the density of the semisimple matrices in Mn{C), we can find ym semisimple so that 
limm^.oo Vm = ol + w. Hence x = limm_!.oo ( ^ ^(T ) ' Notice that the latter matrix is 
semisimple. Indeed, if y E M„(C) is semisimple and gyg~^ — d is a diagonal matrix, then 



g 0\ /O ^\ /g ^1 I = I , f I is also diagonalizable. 

GL„(M) X GL„(1 



There arc three cases to consider. 



(I) The analogue of the case of GL„(C) x GL„(C), with C replaced by M. One proceeds 
as above. 

(II) V = Vo X Vt with V, = ^v^,k (« e Z/2Z). 
Xs = Xs{a) = f j with a e ]R\ {0}. 



Xn = Q j with w, w' e M„ 

From XnXs = XsXn we obtain w' = —w. Hence x = Xs + Xn 



al + w 
-al — w 
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Let Um G M„(M) be semisimple and so that liiiim-^oo 1/m = al + w. Then x 

le. 

y\ fg-' 



Um^^oo ( ^ ^iT ) ■ Notice that the latter matrix is semisimple. Indeed, over 



C, if gyg ^ — d is a diagonal matrix, then ^ 



y oj \o i-g)- 



1 - 



) diagonalizable. 



(Ill) V = Vo X Vx with V„ = ELi(K^a,fe + ^v',,k) (« e Z/2Z). 

Xs = Xs{a) = where = diag(a, . . . , a) is a block diagonal matrix with 

equal 2x2 diagonal blocks a = (^^^ ^ , P^l ^ We can assume (3 and 
7 7^ 0, otherwise we are reduced to the previous cases. 
Xn = (JIj, with w,w' e M„(M). 

From XnXg = XgXn we obtain w'A = Aw and Aw' = wA. This implies that 
w' — A'^wA and A'^w — wA^. Notice that 

where j ~ (^^i • Write w = (wrs) where each Wrs is a 2 x 2 block. As 
A'^w — wA?, we have that Jw = wJ where J = diag(j, ■ ■ ■ Hence Jwrs = WrsJ, 

Prs Irs 



^ Ifrs Prs 

conclusion follows by the same argument as before. 
S = GL„(H) xGL„(H): 



. Therefore w commutes with ^4, i.e. w' — w. The 



There is only one case, and it is as (III) for GL„(R) x GL„ 
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Appendix D 

Recall the function In, (66). Here we verify the following elementary lemma. 
Lemma D.l. For N >0, 



Furthermore, 



poo 2 

/ (1 + (0^ + 0-2)^2)-^ da/a <-7jvW > 0). (D.l) 

Jo ^ 

POO 2 

/ (l + a262 + a-V)i2)-^da/a< {b,c>0). (D.2) 

Jo ^ 

Proof. Since the measure da/ a is invariant under the substitution, a ^ at, t > 0, (D.2) 
follows from (D.l). Since a^ + a^ > 2 > 1, the left hand side of (D.l) is dominated by 

poo 

(1 + t2)-^/2 / (1 + (^2 + a-2)t2)-^/2 da/a (D.3) 
Jo 

/CO 
{l + iatfy'^da/a 

/oo 
(l + a2)-^/2da/a. 

If t > 1 then the last expression in (D.3) is less or equal to 

/oo POO r\ 

(1 + a^)-^/^ da/ a < r^2 a-^-^ da = ^r^. 

If < t < 1 then the last expression in (D.3) is dominated by 

/oo 
(l + a^)-^/^ da/a 

/oo /"^ 2 2 

(1 + a^)-^/2 da/a + 2 J da/a < ;^ - 2 ln(i) ^ —{1 - N ]n{t)). 



□ 
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